Build a robots.txt file and test it.
Choose what search engines and AI crawlers may read, then copy or download the file and test any path against it. Free, no account, unlimited.
Free, no account, unlimited. The file is built in your browser and nothing is sent anywhere.
- Rules for all crawlers
- 18 AI crawlers, one switch each
- Training and AI search kept apart
- Copy or download the file
- Built in tester for any path
What you get
A file you can upload, and proof it does what you meant.
The generator writes a clean robots.txt from your choices, sums up what it allows and blocks, and lets you test paths against it before it goes live.
robots.txt · example.com
ready
- all crawlersallowed
- disallowed paths/admin/, /cart
- training crawlers8 of 8 blocked
- AI answer agents10 of 10 allowed, still cited
- sitemaplisted
- test /admin/ as Googlebotblocked by line 2
Why it matters
One small file decides what gets read.
Search engines and AI crawlers check robots.txt before they request a page. A good file points them at your best pages. A bad one can hide your whole site.
Crawling
Keep crawlers out of clutter
Admin screens, carts and internal search results waste crawl time. A disallow rule keeps crawlers on the pages you want found.
AI visibility
Choose what AI may learn and cite
Training crawlers and AI search agents use separate user agents. You can opt out of training and still be read, cited and linked in AI answers.
Safety
Avoid blocking your whole site
One wrong line, such as Disallow: / for every user agent, hides every page from search. Testing paths before you upload catches it.
Who it's for
Built for the people who run websites.
01
Site owners
Get a correct robots.txt without learning the syntax, and decide in one place how AI tools may use your content.
02
SEO specialists
Draft rules for a client site, test the paths that matter, and hand over a file that will not block pages by accident.
03
Developers
Generate the file for a new build or a staging site, download it, and drop it in the public root.
04
Publishers
Opt out of AI training while staying visible in ChatGPT search, Claude and Perplexity answers that send readers back.
How it works
From choices to a tested file in a minute.
01
Set the basics
Pick allow or block for all crawlers, list the paths to keep private, and add your sitemap URL and a crawl delay if you want them.
02
Choose your AI crawlers
Allow or block each one, or use a shortcut. Training crawlers and AI answer agents are listed apart, so the tradeoff stays clear.
03
Copy, test and upload
Copy or download robots.txt, test any path against it as any crawler, then upload it to the root of your site.
FAQ
Questions, answered.
It is a plain text file at the root of your site, such as example.com/robots.txt, that tells crawlers which paths they may request. Search engines and most AI crawlers read it before they crawl anything else.
Yes. It runs in your browser, needs no account and has no limit. The file is built on your device, so nothing you enter is sent to us.
Upload it to the root of your site, so it loads at yourdomain.com/robots.txt. The name must be lowercase, and each subdomain needs its own file. Crawlers ignore a robots.txt placed in a subfolder.
No. It controls crawling, not indexing. Google can still index a blocked URL without its content if other pages link to it. To keep a page out of results, add a noindex tag and leave the page crawlable so Google can see the tag.
It depends on what you want. Blocking training crawlers such as GPTBot, ClaudeBot or Google-Extended keeps your pages out of future model training. Blocking AI search and assistant agents such as OAI-SearchBot, Claude-SearchBot or PerplexityBot also keeps your pages out of the answers those tools give, and out of the links they show. Many sites block training and allow search.
No. Google says Google-Extended does not affect inclusion in Google Search, and AI Overviews are part of Search, which Googlebot crawls. Blocking Googlebot would remove your site from Google Search altogether.
No. It is a request, not a lock. Major search engines and most named AI crawlers follow it, but some, such as Bytespider, have been reported to ignore it, and Perplexity says visits a user asks for may not follow it. Protect anything private with a login or server rules.
It asks crawlers to wait the given number of seconds between requests. Bing and some other crawlers read it. Google ignores it and sets its own crawl rate, so leave it empty unless your server struggles under crawler traffic.
It reads your generated file the way Google documents: a crawler follows the group that names it, or the * group if none does, and within that group the longest matching rule wins, with allow winning a tie. You see whether the path is allowed and which line decided it. It tests the file you built here, not the live file on your site.
Yes. A Sitemap line is a simple way to help every crawler find your sitemap. Use the full URL, including https://. You can still submit the sitemap in Google Search Console and Bing Webmaster Tools as well.
Free SEO tools