Webflow robots.txt generator for AI crawlers

Webflow exposes robots.txt as an editable field in your site's SEO settings: what you paste is what your published site serves. That makes it one of the simpler platforms on this list, with one caveat worth knowing: the field is the ONLY root text file you control. There is no general file upload to the domain root.

Runs in your browser. Your domain is never sent to us, and nothing is stored.

Type your domain and the file appears here.

Applying this on Webflow

  1. Generate the AI-crawler rules with the tool above.
  2. Paste them into your site's robots.txt field in the SEO settings. If you already have rules there, add the User-agent blocks rather than replacing rules you rely on.
  3. Publish the site. Webflow serves settings changes on publish, not on save.
  4. Re-read yoursite.com/robots.txt on the live domain and confirm the crawlers appear.

llms.txt on Webflow

Webflow does not offer arbitrary root file uploads, so a native llms.txt at your domain root is not available; robots.txt is special-cased through its settings field. If you proxy your Webflow site behind your own edge, you can serve llms.txt from the proxy, and our validator will tell you whether it returns real markdown. llms.txt Validator · llms.txt Generator

Can AI crawlers read your Webflow content at all?

Published Webflow sites serve fully rendered HTML, so your designed content is present for any crawler allowed to fetch it. Content pulled in client-side through custom code is the exception: check those pages with the server-side rendering checker. Server-Side Rendering Checker

Questions

Where do I put robots.txt in Webflow?
In the site's SEO settings, as text in the robots.txt field. There is no file to upload; the field's contents are served at your domain root after you publish.
Why is my robots.txt change not live on Webflow?
Settings take effect on publish. Save, publish the site, then re-read yoursite.com/robots.txt on the live domain rather than the preview.
Can Webflow serve an llms.txt file?
Not natively: robots.txt is the one root text file the platform exposes. Serving llms.txt requires fronting the site with your own proxy or edge layer.

By platform

The file is half the story. Whether AI engines then read and cite your Webflow site is the half you cannot see from your own dashboard. Run the free AI visibility audit