Skip to content

Free robots file generator

Free robots.txt generator with AI crawler rules and a tester.

Create a robots.txt file without learning the syntax. Pick what search engines and AI crawlers may read, add your sitemap, then copy or download the file and test any path against it.

  • No sign-up needed
  • Built in your browser
  • 18 AI crawlers

One per line. Ignored when all crawlers are blocked.

Try an example

AI crawlers

Model training

Blocking these keeps your pages out of future training data. It does not remove you from AI answers.

  • GPTBot OpenAI

    Collects pages to train OpenAI's models.

  • ClaudeBot Anthropic

    Collects pages to train Anthropic's Claude models.

  • Google-Extended Google

    Decides whether Google may train Gemini on your pages. Google Search is not affected.

  • Applebot-Extended Apple

    Decides whether Apple may train its models on your pages. Siri and Spotlight are not affected.

  • CCBot Common Crawl

    Builds the open Common Crawl dataset many AI models train on.

  • Bytespider ByteDance

    Collects pages to train ByteDance's models. Reported not to always follow robots.txt.

  • meta-externalagent Meta

    Collects pages to train Meta's AI models and improve products.

  • Diffbot Diffbot

    Turns pages into structured data sold to AI and analytics teams.

AI search and assistants

These find and read pages for answers in ChatGPT, Claude, Perplexity and others. Block them and those answers cannot cite you.

  • OAI-SearchBot OpenAI

    Indexes pages so ChatGPT search can show and link to them.

  • ChatGPT-User OpenAI

    Opens a page when a ChatGPT user asks about it.

  • Claude-SearchBot Anthropic

    Indexes pages so Claude can find and cite them in answers.

  • Claude-User Anthropic

    Opens a page when a Claude user's question needs it.

  • PerplexityBot Perplexity

    Indexes pages so Perplexity can show and cite them.

  • Perplexity-User Perplexity

    Opens a page when a Perplexity user asks. Perplexity says these visits may ignore robots.txt.

  • Meta-ExternalFetcher Meta

    Opens a link when a Meta AI user asks about it.

  • Amazonbot Amazon

    Reads pages for Amazon services, including answers in Alexa.

  • DuckAssistBot DuckDuckGo

    Reads pages for DuckDuckGo's AI assisted answers.

  • MistralAI-User Mistral

    Opens a page when a Le Chat user asks about it.

Free, no account, unlimited. The file is built in your browser and nothing is sent anywhere.

AI crawlers, one switch each
18AI crawlers, one switch each
Model training crawlers
8Model training crawlers
AI search and assistant agents
10AI search and assistant agents
Sign-ups or daily limits
0Sign-ups or daily limits

What you get

A file you can upload, and proof it does what you meant.

The generator writes a clean robots.txt from your choices, sums up what it allows and blocks, and lets you test paths against it before it goes live. It all runs on your device, so nothing you enter is sent to us.

    • User-agent: *
    • Disallow: /admin/
    • Disallow: /cart
    • User-agent: GPTBot
    • Sitemap: https://example.com/sitemap.xml

    Create a robots.txt file in four choices

    Allow or block all crawlers, list the paths to keep them out of, add your sitemap URL and, if your server needs it, a crawl delay. The robot text generator writes the lines for you.

    • GPTBot
    • ClaudeBot
    • Google-Extended
    • CCBot

    Block AI training crawlers

    Keep your pages out of future model training, one crawler at a time or all eight at once.

    • OAI-SearchBot
    • Claude-SearchBot
    • PerplexityBot

    Stay readable to AI search

    Answer agents are listed apart from training crawlers, so you can opt out of training and still be cited in AI answers.

  • 20

    User agents to test as

    Test any path before you upload

    Type a path, pick a user agent and see whether it's allowed, with the line that decided it.

    • Training crawlers blocked8 of 8
    • AI answer agents blocked0 of 10

    Example settings

    A summary, then copy or download

    Before you copy the file, a summary shows what it blocks, so a stray rule stands out.

Build your robots.txt file now.

Set your rules, choose your AI crawlers and get a file you can upload today. Free, no account, no limit.

Generate robots.txt

Why it matters

One small file decides what gets read.

  • Keep crawlers out of clutter

    Admin screens, carts and internal search results waste crawl time. A disallow rule keeps crawlers on the pages you want found.

  • Choose what AI may learn and cite

    Training crawlers and AI search agents use separate user agents. You can say no to one and yes to the other.

  • Avoid blocking your whole site

    One wrong line, such as Disallow: / for every user agent, hides every page from search. Testing paths before you upload catches it.

How to create a robots.txt file

From choices to a tested file in a minute.

  1. 1

    Set the basics

    Pick allow or block for all crawlers, list the paths to keep private, and add your sitemap URL and a crawl delay if you want them.

  2. 2

    Choose your AI crawlers

    Allow or block each one, or use a shortcut. Training crawlers and AI answer agents are listed apart, so the tradeoff stays clear.

  3. 3

    Generate, test and upload

    Generate the file, test the paths that matter, then copy or download it and upload it to the root of your site.

Read it honestly

What robots.txt can and can't do for you.

A robots.txt file is a set of requests to crawlers. It's useful, and it has limits worth knowing before you rely on it.

What the generator does

  • Writes a valid file from your choices: a default rule for every crawler, disallowed paths, a sitemap line, a crawl delay, and its own group for the AI crawlers you treat differently from the rest.
  • Tests a path the way Google documents: a crawler follows the group that names it, or the * group, and the longest matching rule wins, with allow winning a tie.
  • Keeps training crawlers and AI search agents apart, so blocking one doesn't quietly block the other.

What it can't do

  • Force a crawler to obey. Major search engines and most named AI crawlers follow robots.txt, but it's a request, not a lock. Protect private pages with a login.
  • Keep a page out of Google. Robots.txt controls crawling, not indexing. Use a noindex tag for that, and leave the page crawlable.
  • Test the file live on your site. The tester checks the file you built here. Google also ignores crawl delay and sets its own pace.

Who it's for

Built for the people who run websites.

  • Site owners

    Get a correct robots.txt without learning the syntax, and decide in one place how AI tools may use your content.

  • SEO specialists

    Draft rules for a client site, test the paths that matter, and hand over a file that won't block pages by accident.

  • Developers

    Generate the file for a new build or a staging site, download it, and drop it in the public root.

  • Publishers

    Opt out of AI training while staying visible in ChatGPT search, Claude and Perplexity answers that send readers back.

FAQ

Questions, answered.

Free SEO tools

File uploaded? Test the live one on your site.