Tracked AI crawler family
Common Crawl
CCBot crawls the open web into a public dataset that many model builders train on, upstream of any single assistant.
Feeds: an open dataset many models train on.
The bots in this family
CCBot fetches pages into Common Crawl, an open dataset that a large number of model builders train on. Blocking it removes you from that dataset, which sits upstream of many models rather than feeding one named assistant.
| Bot | robots.txt token | Purpose |
|---|---|---|
| CCBot | ccbot | Training |
How we observe it
- Access. We fetch your robots.txt and evaluate it per named token using RFC 9309 matching. Each bot reads allowed, blocked, or unknown. If robots.txt could not be fetched, every row is unknown and nothing is guessed.
- Activity. Real requests observed on your own server or edge logs, classified by User-Agent against the same catalog, then reconciled path by path against your robots.txt. A violation is claimed only where an explicit disallow rule exists for that bot and path.
The same catalog powers the free crawler check tool and the Analyze crawler panels.
What we do not measure here
Stating the limit is the point. A number is only trustworthy next to the boundary of what it does not cover.
- Which downstream models actually used the data. That is not observable from your site, so we never attribute a specific model to a CCBot fetch.
- Any citation or visibility score in a named assistant. Common Crawl is a dataset, not an engine we sample.
- Crawler identity is self-reported. A User-Agent string is claimed, not cryptographically verified, so a spoofed agent is possible and we treat the identity as claimed, never proven.
Every surface we measure
Engines we sample
Crawler families we observe