Tracked AI crawler family
Applebot-Extended
A robots.txt control on top of Applebot that governs whether Apple may use already-crawled content to train Apple Intelligence.
Feeds: Apple Intelligence training.
The bots in this family
Applebot-Extended does not fetch pages itself. It is a robots.txt token that tells Apple whether content Applebot already crawled may be used to train Apple Intelligence. Blocking it does not remove you from Siri or Spotlight suggestions; it withholds you from that training use.
| Bot | robots.txt token | Purpose |
|---|---|---|
| Applebot-Extended | applebot-extended | Training |
How we observe it
- Access. We fetch your robots.txt and evaluate it per named token using RFC 9309 matching. Each bot reads allowed, blocked, or unknown. If robots.txt could not be fetched, every row is unknown and nothing is guessed.
- Activity. This family is a robots.txt control, not a fetching agent, so it sends no distinct requests to observe. We report its access directive and stop there rather than imply a server-hit reconciliation that does not exist.
The same catalog powers the free crawler check tool and the Analyze crawler panels.
What we do not measure here
Stating the limit is the point. A number is only trustworthy next to the boundary of what it does not cover.
- Server-hit activity, because a control token sends no requests of its own. We observe the robots.txt directive only.
- Any Apple Intelligence visibility score. Apple is not one of our measured engines, so there is no citation number here.
- Whether Apple honored the directive internally, which is not observable from outside Apple.
Every surface we measure
Engines we sample
Crawler families we observe