Technical SEO audits, inside Claude

The technical audit I was writing for clients in 2006 is, structurally, the audit most agencies still sell today: a crawler runs, a template fills, a 60-page PDF lands. Everything a crawler could find, in severity order, with no idea which findings are worth money and which are cosmetic.

What has changed is what's possible. Google gives every site owner a complete record of its search reality - which queries, which pages, how many impressions, where you ranked. Your crawl tells you what your site says; Search Console tells you what Google did about it. In my experience, the gap between those two datasets is where nearly all of the recoverable traffic hides. One idea underneath the whole build: your crawl is intent, Search Console is reality, and the money is where they diverge.

A flat crawler tells you a page 404s. Useful, but only just. This tells you the 404 is draining 15% of your homepage's internal PageRank, that the page used to earn 10,000 clicks a month - and it writes the 301 rule to fix it. Severity is what crawlers sell you. Yield is what moves the numbers.

Ninety-three checks over the joined data, every finding ranked by the clicks it could recover per hour of developer effort, every fix written for you - paste-ready redirects, JSON-LD, internal-link plans, grounded content briefs. The families: crawlability and indexation (with the reason every URL isn't indexable, not just the fact); on-page and structured data, validated against ~30 rich-result types; trends over your GSC history (pages losing clicks, vanished queries, content decaying year-on-year); the merged questions no single dataset can answer - cannibalisation, striking distance, ghost pages, internal authority wasted on no-click pages; and AI-search readiness, because the web is quietly growing a second audience and almost no SEO tool checks any of it.

Every check is labelled deterministic or judgement, and the judgement ones are off by default - a wrong finding is worse than no finding at all, so the heuristics have to ask permission.

The crawl itself is where audits usually go wrong, and I've spent enough of my career cleaning up after crawlers that fooled themselves. This one discovers pages three ways - links, sitemaps, and every URL Google is already sending traffic to - so coverage stops depending on your sitemap being honest. It refuses to store an off-site redirect as a page, uses GET rather than HEAD because HEAD lies, respects robots.txt properly, and gets its speed from HTTP/2 and connection reuse rather than from hammering your server. Being a good guest is the point.

And with DataForSEO connected, it replaces the subscription tools for competitor work: any domain's visibility, top pages and ranked keywords (including AI Overview citations), content gaps, topic gaps. Pay-as-you-go - a full competitor topic-gap run costs about 13 cents, and my own usage runs to a few dollars a month against the $144/month it displaced.

My view is a simple one: an agency can hugely reduce the workload on its team with better crawl and SEO API integration. The audit most agencies still sell is mostly collection - a junior exports Search Console, runs the crawler, stitches the two together in a spreadsheet, and the senior who could actually read it doesn't see it for two days. That's two days billed against gathering, not thinking.

This collapses the collection to a conversation. One command pulls the crawl, the Search Console history and the market data into a single database per client; the next asks it what's wrong, in priority order, with the fix already written. The junior's two days become twenty minutes, and the senior gets the finding while it still matters. Run it across a portfolio of clients and the saving compounds every month.

It also replaces a stack of per-seat subscriptions. The market data comes from DataForSEO, pay-as-you-go, so competitor visibility, ranked keywords, content gaps and AI Overview citations cost cents a client rather than a licence a head. For an agency running the numbers across a team, that's the SEO tool bill going from a fixed monthly line to a few dollars of usage.

On one property, seeding the crawl from Search Console URLs took coverage of GSC-known pages from 29% to 70% - every one of those extra pages is a page a conventional crawl silently missed, several earning traffic with no internal links pointing at them at all. On the same property, incremental sync turned a 33-minute refresh into 19 seconds, which is the difference between "audit quarterly" and "audit whenever you're curious".

When the audit finds something serious, this is the practice that deals with it. The sharpest recent case: my own job platform, YubHub, watched rankings slide for weeks with no visible cause. The deploy pipeline had silently frozen nine days earlier - the site had crossed its host's 20,000-file deployment ceiling and couldn't publish anything, while a redirect loop quietly told Google to drop pages. Root-caused and cleared in a single session; a 356-URL statistical audit afterwards came back with zero residual errors. The site ships with a build-failure alarm now, so a silent freeze can't happen twice.

Live, open source, on npm - the repo and full manual are public. You type "run an SEO audit" into Claude and it happens; the dashboard screenshot on this page is my own site's real data. I run this on everything I operate.