Tutorial: Running a Complete Technical SEO Audit with Screaming Frog

Screaming Frog SEO Spider is the most popular desktop-based crawler for technical SEO audits. This tutorial walks through configuring and running a comprehensive technical audit analyzing results and creating an actionable report.

Step 1: Configuration

Before crawling configure Screaming Frog properly. Open Configuration then Spider and set crawl depth to unlimited for full site coverage. Check CSS JavaScript and images under Crawl. Enable Store Raw HTML for content analysis. Set user agent to Googlebot for accurate simulation. Configure crawl speed based on your server capacity.

Step 2: robots.txt and Sitemap Setup

Upload your robots.txt file for accurate crawl simulation. Add your XML sitemap URL. This allows Screaming Frog to compare sitemap URLs against crawlable URLs and identify discrepancies.

Step 3: Run the Crawl

Enter your website URL and click Start. For large sites over 100000 pages consider crawling sections separately using the Include and Exclude path filters. Monitor the crawl progress and watch for error spikes that might indicate server issues.

Step 4: Analyze Results

Response Codes

Check the Response Codes tab for 404 errors that need fixing or redirecting, 301 and 302 redirects including chains, 500 server errors indicating backend problems, and soft 404s.

Page Titles and Meta Descriptions

Review the Page Titles tab for missing titles, duplicate titles, titles that are too long or too short. Similarly check the Meta Description tab for missing duplicate or poorly sized descriptions.

Canonical Tags

In the Canonical tab identify pages with missing canonical tags, pages where canonical points to a different URL, canonical chains, and verify self-referencing canonicals.

Internal Links

The Internal tab shows orphan pages with no internal links, pages with very few internal links, link depth analysis showing pages too many clicks from the homepage, and internal link distribution across important versus unimportant pages.

Step 5: Export and Report

Export each relevant tab as CSV for further analysis. Create a prioritized issue report with issue category and description, affected URLs count, severity level, recommended fix, and estimated impact. Use the Bulk Export feature to generate comprehensive data files.

Step 6: Create an Action Plan

Prioritize fixes based on impact: First fix crawl-blocking issues like robots.txt blocking important content. Then resolve indexation problems like incorrect canonical tags. Next address site architecture issues. Follow with content optimization. Finally handle performance improvements.

Conclusion

A Screaming Frog technical audit provides the data foundation for prioritized SEO improvements. Run audits quarterly for ongoing sites immediately after major site changes and before launching new website sections.