Generate a robots.txt that lets AI engines in
Default-allow is silent, and silence is not the same as an invitation. This names each live AI crawler and allows it at your root, so your policy is explicit and auditable rather than implied. The list comes from the same catalog our checkers measure against, so it cannot drift from what we report.
Runs in your browser. Your domain is never sent to us, and nothing is stored.
Type your domain and the file appears here.
Questions
- My robots.txt already allows everyone. Why name them?
- Mostly for you, not for them. A named list is a policy you can read six months later and a diff you can review, instead of an assumption. It also makes deliberate exclusions visible: if you decide to block a training crawler, that decision is written down rather than inferred from an absence.
- Why is Bytespider not in the list?
- Because naming a crawler Allow is an explicit invitation, and whether ByteDance may take your content is your call rather than ours. We only name the crawlers in our measured catalog. Nothing here blocks Bytespider either: the wildcard rule still applies to it.
- Will this delete my existing rules?
- It generates a fresh file, so treat it as a starting point rather than a replacement. If you already have Disallow paths, keep them and paste these User-agent blocks in alongside.
- Is this free?
- Yes, and there is no signup. It runs in your browser, so your hostname is never sent to us.
Check something else
- Is GPTBot blocked on your site?Tests the GPTBot user-agent specifically, which builds the index the model answers from.
- Is ClaudeBot blocked on your site?Tests the ClaudeBot user-agent specifically, which builds the index the model answers from.
- Generate an llms.txt for your siteWrites the navigation file that points models at your best pages. The validator grades an llms.txt you already have; this gives you the first draft.
- Generate Organization schema for your homepageWrites the identity block. The other two govern whether engines may read you and where they should look; this one says who they are reading.