Tracked AI crawler family
OpenAI crawlers
The OpenAI bots that can reach your pages, what each one is for, and how we observe their access and activity from your own robots.txt and server logs.
Feeds: ChatGPT.
The bots in this family
GPTBot indexes pages for OpenAI, OAI-SearchBot builds the index behind ChatGPT Search, and ChatGPT-User fetches a page live when someone asks ChatGPT to open your link. Block one in robots.txt and that path into ChatGPT closes, which is upstream of ever being cited.
| Bot | robots.txt token | Purpose |
|---|---|---|
| GPTBot | gptbot | Indexing |
| OAI-SearchBot | oai-searchbot | Search |
| ChatGPT-User | chatgpt-user | Live browsing |
How we observe it
- Access. We fetch your robots.txt and evaluate it per named token using RFC 9309 matching. Each bot reads allowed, blocked, or unknown. If robots.txt could not be fetched, every row is unknown and nothing is guessed.
- Activity. Real requests observed on your own server or edge logs, classified by User-Agent against the same catalog, then reconciled path by path against your robots.txt. A violation is claimed only where an explicit disallow rule exists for that bot and path.
The same catalog powers the free crawler check tool and the Analyze crawler panels.
What we do not measure here
Stating the limit is the point. A number is only trustworthy next to the boundary of what it does not cover.
- Crawler identity is self-reported. A User-Agent string is claimed, not cryptographically verified, so a spoofed agent is possible and we treat the identity as claimed, never proven.
- Unknown bots. Anything outside the named catalog is dropped at ingest, because the claim is about named AI crawlers, not everything that wanders past.
- A fetch is not a citation. Observing GPTBot read your page does not mean ChatGPT cited you. Citation is measured separately on the ChatGPT engine page.
Access is a precondition for being cited, not the citation itself. Whether ChatGPT then cites you is measured separately on the ChatGPT engine page.
Every surface we measure
Engines we sample
Crawler families we observe