What AI crawlers actually did

What it is

The crawler traffic your own infrastructure saw, joined against your current robots.txt. Robots.txt is a promise about what is permitted; this is what happened. Each named crawler lands in one of four states: violating (seen fetching paths it is disallowed from), consistent, dead allowance (allowed in and never seen), or blocked and absent.

When to use it

When you want to know whether AI engines are reaching your pages at all, whether the allowances sitting in your robots.txt are doing anything, and whether something is fetching paths you closed off.

Step by step

1. 1Open Settings and then SEOAIO Tracking, and install the crawl collector. AI crawlers do not run JavaScript, so the page tag can never see them; the collector sits at your server or edge instead. There is a ready middleware.ts for Vercel and Next.js, a Cloudflare Worker, and a plain POST for anything else.
2. 2Open Analyze and scroll to “What AI crawlers actually did”.
3. 3Read the three headline figures: crawler hits over the last 30 days, claimed-identity violators, and dead allowances.
4. 4Work down the per-bot list. A violating row names its worst disallowed path and the hits on it; a dead-allowance row is a door you left open that nobody walked through.

What you will see

A hits total for the window, a violator count, a dead-allowance count, and one row per named crawler showing its engine, its purpose (indexing, search, live user fetch, or training), and what it did. With no collector reporting, the panel reads Not measured and links to the install page rather than drawing an empty chart that looks like zero traffic.