Robots.txt Generator
control what crawlers fetch
Build a robots.txt file without memorising the syntax: start from a preset, add the folders to keep crawlers out of, list your sitemap and decide what to do about AI crawlers.
- Free
- No sign-up
- Presets to start from
- Sitemap lines
- Optional AI crawler blocking
Start from
Runs entirely in your browser — nothing you enter is sent to us.
How it works
How to use the Robots.txt Generator
-
Pick a starting point
Allow everything, block everything, or start from a typical website setup.
-
Add your rules
List the paths to disallow, the exceptions to allow, and your sitemap URL.
-
Upload the file
Download robots.txt and put it at the root of your domain, e.g. example.com/robots.txt.
Where does robots.txt go?
At the root of the host it applies to: https://example.com/robots.txt. Each subdomain needs its own file, and crawlers ignore robots.txt files in subfolders.
Does Disallow keep a page out of Google?
No. Disallow stops crawling, not indexing: a blocked URL can still appear in results if other sites link to it. To keep a page out of search, allow crawling and add a noindex robots meta tag.
Can robots.txt block AI crawlers?
Most AI companies say their crawlers obey robots.txt, so you can disallow user agents like GPTBot, ClaudeBot, CCBot, Google-Extended and PerplexityBot. It is a request, not an enforcement: crawlers that ignore the file are not stopped by it.
More free tools
Other tools you might need
Share it with a short link
Put a Letlink short link on the page you just checked and see who clicks it, where they come from and on which device.
Last updated by the Letlink team.