Claude Skill · Technical Foundation

Screaming Frog Crawl Analysis

Screaming Frog gives you 40 columns and 50,000 rows. The skill is knowing which six columns answer which question — and how to cross-reference exports so the findings are causal, not just descriptive.

Language · Markdown License · MIT Type · Claude Skill

View on GitHub north_east Fork it fork_right

01 · The Problem

Why This Exists

A list of "pages with missing titles" is a filter, not an audit. Anyone can sort a column. The value is in knowing which finding causes the others.

A noindexed page with a missing title tag does not have a title-tag problem. Resolve indexability as a logic tree and report the first failing gate — everything downstream is noise.

And prioritization is only credible when findings are weighted by traffic and links. A broken canonical on a zero-traffic page is not a priority, and saying so out loud is what builds trust with a client.

02 · Method

How It Works

01

Parse, don't eyeball

analyze_crawl.py handles the export's quirks — mixed encodings, the header row, Address as key — and returns a structured summary to reason about rather than a raw table to squint at.

02

Resolve indexability as a gate sequence

Status 200 → robots directive → canonical → robots.txt → internally reachable → in sitemap. Report "fails at gate N", never a pile of independent flags.

03

Run the five analyses that matter

Crawl depth, true orphans (traffic or links but zero inlinks), redirect chains resolved to final destination, internal link equity per URL, and duplicate clusters diagnosed as patterns rather than 400 pasted rows.

04

Weight, rank, and hand over

Every finding carries scale, evidence, and business weight. Output includes a dev-ready redirect remediation list and specific internal linking actions — link these money pages from those authority pages.

03 · Run It

Two Minutes To First Output

# 1. Clone the bundle
git clone https://github.com/seosdm/seo-skills.git
cd seo-skills

# 2. Export internal_all.csv from Screaming Frog (add all_inlinks.csv if you can)
python skills/screaming-frog-crawl-analysis/scripts/analyze_crawl.py \
  internal_all.csv --inlinks all_inlinks.csv

# 3. Copy skills/screaming-frog-crawl-analysis/SKILL.md into a Claude Project:
Run screaming-frog-crawl-analysis on this export.
GSC data attached for weighting.

No API keys, no setup. The skill is a SKILL.md file plus a worked example — it runs anywhere Claude does.

04 · Proof

Sample Output

Indexability by First Failing Gate Example · anonymized

61% Indexable

1,204 Fail · Canonical

318 Fail · Depth >4

27 True Orphans

Verdict: faceted navigation on /shop/* is generating 1,204 self-conflicting canonicals — one templating bug, not 1,204 problems. Fixing it recovers 39% of the crawl. The 27 true orphans carry backlinks and are the second move, not the first.

See the full worked example in /examples.

If The Tool Gets You 60% There

I'll Teach You The Rest

Free skill, real judgment behind it. Want it run on your business, or your team taught to run it themselves?

Get This Run On Your Site north_east