Tutorial: Complete Technical SEO Audit with Screaming Frog and Python

This tutorial walks you through conducting a complete technical SEO audit using Screaming Frog for crawling and Python for advanced analysis. By the end, you will have identified and prioritized all technical SEO issues.

Prerequisites

• Screaming Frog SEO Spider (free version handles up to 500 URLs)
• Python 3.11+ with pandas, requests, and beautifulsoup4
• Google Search Console access
• Access to server log files (optional but recommended)

Step 1: Configure Screaming Frog Crawl

Crawl Settings

1. Open Screaming Frog → Configuration → Spider
2. Enable: Crawl canonicalized pages, JavaScript rendering (Chrome Crawler)
3. Set user agent to Googlebot (mobile)
4. Exclude: Images, CSS, JS (focus on HTML pages)
5. Set maximum crawl depth to 10

Custom Extraction

Add custom extractions for:

• H1 text (XPath: //h1)
• Meta robots (XPath: //meta[@name="robots"]/@content)
• Canonical URL (XPath: //link[@rel="canonical"]/@href)
• Schema types (XPath: //script[@type="application/ld+json"])

Step 2: Export and Analyze Crawl Data

Export Key Reports

1. Internal → All (full crawl data)
2. Response Codes → Client Error (4xx)
3. Response Codes → Server Error (5xx)
4. Redirect Chains (3xx)
5. Page Titles → Missing, Duplicate, Over/Under length
6. Meta Description → Missing, Duplicate, Over/Under length

Python Analysis

import pandas as pd

def analyze_screaming_frog_export(filepath):
    df = pd.read_csv(filepath, low_memory=False)
    
    report = {