Free robots file generator
Free robots.txt generator with AI crawler rules and a tester.
Create a robots.txt file without learning the syntax. Pick what search engines and AI crawlers may read, add your sitemap, then copy or download the file and test any path against it.
- No sign-up needed
- Built in your browser
- 18 AI crawlers
Free, no account, unlimited. The file is built in your browser and nothing is sent anywhere.
- AI crawlers, one switch each
- 18AI crawlers, one switch each
- Model training crawlers
- 8Model training crawlers
- AI search and assistant agents
- 10AI search and assistant agents
- Sign-ups or daily limits
- 0Sign-ups or daily limits
What you get
A file you can upload, and proof it does what you meant.
The generator writes a clean robots.txt from your choices, sums up what it allows and blocks, and lets you test paths against it before it goes live. It all runs on your device, so nothing you enter is sent to us.
- User-agent: *
- Disallow: /admin/
- Disallow: /cart
- User-agent: GPTBot
- Sitemap: https://example.com/sitemap.xml
Create a robots.txt file in four choices
Allow or block all crawlers, list the paths to keep them out of, add your sitemap URL and, if your server needs it, a crawl delay. The robot text generator writes the lines for you.
- GPTBot
- ClaudeBot
- Google-Extended
- CCBot
Block AI training crawlers
Keep your pages out of future model training, one crawler at a time or all eight at once.
- OAI-SearchBot
- Claude-SearchBot
- PerplexityBot
Stay readable to AI search
Answer agents are listed apart from training crawlers, so you can opt out of training and still be cited in AI answers.
20
User agents to test as
Test any path before you upload
Type a path, pick a user agent and see whether it's allowed, with the line that decided it.
- Training crawlers blocked8 of 8
- AI answer agents blocked0 of 10
Example settings
A summary, then copy or download
Before you copy the file, a summary shows what it blocks, so a stray rule stands out.
Build your robots.txt file now.
Set your rules, choose your AI crawlers and get a file you can upload today. Free, no account, no limit.
Why it matters
One small file decides what gets read.
Keep crawlers out of clutter
Admin screens, carts and internal search results waste crawl time. A disallow rule keeps crawlers on the pages you want found.
Choose what AI may learn and cite
Training crawlers and AI search agents use separate user agents. You can say no to one and yes to the other.
Avoid blocking your whole site
One wrong line, such as Disallow: / for every user agent, hides every page from search. Testing paths before you upload catches it.
How to create a robots.txt file
From choices to a tested file in a minute.
- 1
Set the basics
Pick allow or block for all crawlers, list the paths to keep private, and add your sitemap URL and a crawl delay if you want them.
- 2
Choose your AI crawlers
Allow or block each one, or use a shortcut. Training crawlers and AI answer agents are listed apart, so the tradeoff stays clear.
- 3
Generate, test and upload
Generate the file, test the paths that matter, then copy or download it and upload it to the root of your site.
Read it honestly
What robots.txt can and can't do for you.
A robots.txt file is a set of requests to crawlers. It's useful, and it has limits worth knowing before you rely on it.
What the generator does
- Writes a valid file from your choices: a default rule for every crawler, disallowed paths, a sitemap line, a crawl delay, and its own group for the AI crawlers you treat differently from the rest.
- Tests a path the way Google documents: a crawler follows the group that names it, or the * group, and the longest matching rule wins, with allow winning a tie.
- Keeps training crawlers and AI search agents apart, so blocking one doesn't quietly block the other.
What it can't do
- Force a crawler to obey. Major search engines and most named AI crawlers follow robots.txt, but it's a request, not a lock. Protect private pages with a login.
- Keep a page out of Google. Robots.txt controls crawling, not indexing. Use a noindex tag for that, and leave the page crawlable.
- Test the file live on your site. The tester checks the file you built here. Google also ignores crawl delay and sets its own pace.
Who it's for
Built for the people who run websites.
Site owners
Get a correct robots.txt without learning the syntax, and decide in one place how AI tools may use your content.
SEO specialists
Draft rules for a client site, test the paths that matter, and hand over a file that won't block pages by accident.
Developers
Generate the file for a new build or a staging site, download it, and drop it in the public root.
Publishers
Opt out of AI training while staying visible in ChatGPT search, Claude and Perplexity answers that send readers back.
FAQ
Questions, answered.
It's a plain text file at the root of your site, such as example.com/robots.txt, that tells crawlers which paths they may request. Search engines and most AI crawlers read it before they crawl anything else.
Choose which crawlers to allow or block, add the paths to keep them out of and your sitemap address, then generate the file. Copy or download the result and upload it to the root of your site.
Yes. It runs in your browser, needs no account and has no limit. The file is built on your device, so nothing you enter is sent to us.
Upload it to the root of your site, so it loads at yourdomain.com/robots.txt. The name must be lowercase, and each subdomain needs its own file. Crawlers ignore a robots.txt placed in a subfolder.
No. It controls crawling, not indexing. Google can still index a blocked URL without its content if other pages link to it. To keep a page out of results, add a noindex tag and leave the page crawlable so Google can see the tag.
It depends on what you want. Blocking training crawlers such as GPTBot, ClaudeBot or Google-Extended keeps your pages out of future model training. Blocking AI search agents such as OAI-SearchBot, Claude-SearchBot or PerplexityBot also keeps your pages out of the answers those tools give, and out of the links they show. Many sites block training and allow search.
No. Google says Google-Extended does not affect inclusion in Google Search, and AI Overviews are part of Search, which Googlebot crawls. Blocking Googlebot would remove your site from Google Search altogether.
Yes. A Sitemap line is a simple way to help every crawler find your sitemap. Use the full URL, including https://. You can still submit the sitemap in Google Search Console and Bing Webmaster Tools as well.