Drupal robots.txt generator for AI crawlers
Drupal core ships a real robots.txt in your web root, and Composer puts it there: it is a scaffold file, and by default every composer install or update copies core's version over whatever you edited. That is the Drupal trap. Your AI-crawler rules can be live today and silently gone after the next core update, unless the project tells the scaffold to append your rules or to leave the file alone.
Runs in your browser. Your domain is never sent to us, and nothing is stored.
Type your domain and the file appears here.
Applying this on Drupal
- Read yoursite.com/robots.txt. If its header comment is Drupal's stock text, you are serving core's scaffold copy.
- Generate the AI-crawler rules with the tool above.
- Save them in a file in your project (for example assets/my-robots-additions.txt) and add an append entry for [web-root]/robots.txt under extra.drupal-scaffold.file-mapping in composer.json. drupal.org recommends append or prepend over replacing the file, so core's own rules keep receiving fixes.
- Run composer install so the scaffold rewrites robots.txt with your additions, deploy, and re-read the live file.
- If you would rather edit robots.txt in the admin UI with the RobotsTxt module, delete or rename the physical file first: the module serves its own version only when no file is in the way.
llms.txt on Drupal
On self-hosted Drupal, llms.txt is a static file you place in the web root next to robots.txt, and the web server serves it like any other file. It is not one of core's scaffold files, so Composer leaves it alone. On SaaS Drupal without server access, such as Acquia Source, whether you can serve a root file at all is something we have not verified: ask the vendor, then check the URL with our llms.txt validator. llms.txt Validator · llms.txt Generator
Can AI crawlers read your Drupal content at all?
Drupal renders pages on the server through its theme layer, so content in a classic Drupal site is in the HTML an AI crawler fetches. The exception is decoupled Drupal, where a JavaScript front end fetches content from Drupal's API in the browser: run the server-side rendering checker on a content page before assuming crawlers see it. Server-Side Rendering Checker
Questions
- Why did my robots.txt changes disappear after a Drupal update?
- robots.txt is a Composer scaffold file, and scaffold files overwrite the target by default. Add your rules through an append entry in composer.json, or map [web-root]/robots.txt to false. drupal.org warns that the second option also stops core's fixes to the file reaching you.
- Should I use the RobotsTxt module or a file?
- Either works on self-hosted Drupal. The module lets you edit the file in the admin UI and per site on a multisite install, and it needs the physical robots.txt deleted or renamed so its own version is served.
- Who can edit robots.txt and llms.txt on Acquia Source?
- We have not verified it. Acquia Source does not allow contributed modules or server access, so the self-hosted methods on this page do not apply there. Read what your domain serves today and ask Acquia which of these files you control.
By platform
The file is half the story. Whether AI engines then read and cite your Drupal site is the half you cannot see from your own dashboard. Run the free AI visibility audit