Is CCBot blocked on your site?
CCBot is the Common Crawl (feeds many models) crawler that collects pages used to train the model. If your robots.txt blocks it, that engine cannot read your pages, and a page it cannot read is a page it can never cite.
Free, no sign-up. We fetch one public file and read it. Nothing is stored.
Questions
- What does CCBot do?
- CCBot is used by Common Crawl (feeds many models). It collects pages used to train the model.
- Should I block CCBot?
- That is a real trade-off, not a bug to fix. Blocking it keeps your content out of model training, but can also reduce how well the assistant knows you. Blocking a TRAINING crawler is a defensible choice; blocking a SEARCH or live-browsing crawler is what actually costs you citations.
- Is this check free?
- Yes, and there is no signup. We fetch one public file, robots.txt, and read the rules for this user-agent. You see the full result, including every failure we find.
Check something else
- Is Google-Extended blocked on your site?Tests the Google-Extended user-agent specifically, which collects pages used to train the model.
- Is Applebot-Extended blocked on your site?Tests the Applebot-Extended user-agent specifically, which collects pages used to train the model.
- Is GPTBot blocked on your site?Tests the GPTBot user-agent specifically, which builds the index the model answers from.
- Is your sitemap.xml reachable?Fetches /sitemap.xml and confirms it is served as XML, catching the case where a site returns its HTML app shell instead.