Tracked AI crawler family
Perplexity crawlers
The Perplexity bots that can reach your pages, and how we observe their access and activity. Whether Perplexity then cites you is measured separately.
Feeds: Perplexity.
The bots in this family
PerplexityBot indexes pages for Perplexity, and Perplexity-User fetches a page live when someone's query pulls your link into an answer. Block either in robots.txt and Perplexity loses that route to your content.
| Bot | robots.txt token | Purpose |
|---|---|---|
| PerplexityBot | perplexitybot | Indexing |
| Perplexity-User | perplexity-user | Live browsing |
How we observe it
- Access. We fetch your robots.txt and evaluate it per named token using RFC 9309 matching. Each bot reads allowed, blocked, or unknown. If robots.txt could not be fetched, every row is unknown and nothing is guessed.
- Activity. Real requests observed on your own server or edge logs, classified by User-Agent against the same catalog, then reconciled path by path against your robots.txt. A violation is claimed only where an explicit disallow rule exists for that bot and path.
The same catalog powers the free crawler check tool and the Analyze crawler panels.
What we do not measure here
Stating the limit is the point. A number is only trustworthy next to the boundary of what it does not cover.
- Crawler identity is self-reported. A User-Agent string is claimed, not cryptographically verified, so a spoofed agent is possible and we treat the identity as claimed, never proven.
- Unknown bots outside the named catalog, which we drop rather than count.
- A fetch is not a citation. Access is a precondition for being cited, not the citation itself, which we measure on the Perplexity engine page.
Access is a precondition for being cited, not the citation itself. Whether Perplexity then cites you is measured separately on the Perplexity engine page.
Every surface we measure
Engines we sample
Crawler families we observe