Tracked AI crawler family
Anthropic crawlers
The Claude bots that can reach your pages, what each one does, and how we observe their access and activity. Claude is not one of our measured engines, and this page says so.
Feeds: Claude.
The bots in this family
ClaudeBot indexes pages for Anthropic, Claude-SearchBot builds the index behind Claude's search, and Claude-User fetches a page live when someone asks Claude to open your link. Blocking one closes that path into Claude.
| Bot | robots.txt token | Purpose |
|---|---|---|
| ClaudeBot | claudebot | Indexing |
| Claude-User | claude-user | Live browsing |
| Claude-SearchBot | claude-searchbot | Search |
How we observe it
- Access. We fetch your robots.txt and evaluate it per named token using RFC 9309 matching. Each bot reads allowed, blocked, or unknown. If robots.txt could not be fetched, every row is unknown and nothing is guessed.
- Activity. Real requests observed on your own server or edge logs, classified by User-Agent against the same catalog, then reconciled path by path against your robots.txt. A violation is claimed only where an explicit disallow rule exists for that bot and path.
The same catalog powers the free crawler check tool and the Analyze crawler panels.
What we do not measure here
Stating the limit is the point. A number is only trustworthy next to the boundary of what it does not cover.
- Your citation share inside Claude. Claude is not one of the engines we sample, so this page reports crawler access and activity only, never a Claude visibility score.
- Crawler identity is self-reported. A User-Agent string is claimed, not cryptographically verified, so a spoofed agent is possible and we treat the identity as claimed, never proven.
- A fetch is not a citation. Observing ClaudeBot read your page tells you it can reach you, not that Claude cited you.
Every surface we measure
Engines we sample
Crawler families we observe