Robots.txt tester: check if a URL is allowed or blocked
Enter a page and pick a crawler. The tester reads the site's live robots.txt and tells you whether that crawler may crawl the page, which rule on which line decided it, and which lines in the file crawlers will skip. You get 10 free tests a day, no account needed.
- Allowed or blocked, per URL
- The deciding rule and its line
- Googlebot, Bingbot and 18 AI crawlers
- Unreadable lines flagged
- Sitemaps listed
- crawlers tested per URL
- 20
- of the file read
- 500 KiB
- free tests
- 10 a day
- needed to test
- No account
What the robots.txt checker takes and gives back
- You enter
- One page URL or a bare domain, up to 2,000 characters, and one crawler from the list
- Crawlers
- Googlebot, Bingbot and 18 AI crawlers
- File it reads
- /robots.txt at the root of the URL's host, over https
- How much it reads
- The first 500 KiB, the same amount Google reads
- You get back
- Allowed or blocked, the file status, how many groups and sitemaps the file has, and the rule and line that decided it
- Also shown
- The same URL tested for all 20 crawlers, the numbered file with the deciding line marked, and a list of problems
- Price
- Free, 10 tests a day, no sign-up
How to test robots.txt in three steps
Test robots.txt online from your browser. There is nothing to install and no login.
01
Enter a URL
Paste a page address such as example.com/blog/post, or just the domain to test the homepage. You can leave out https://.
02
Pick a crawler
Choose Googlebot, Bingbot or one of the AI crawlers, then select Test URL.
03
Read the verdict
You see allowed or blocked, the rule that decided it, the whole file with that line marked, and any lines crawlers cannot read. Fix the file, then run the test again.
What each robots.txt test shows you
The rule that decided it
The verdict names the exact Allow or Disallow line and its line number, so you know which line to change. When no rule matches, it says so.
Every crawler at once
One test of a URL fills a table of all 20 crawlers with their verdict and deciding rule. You can see at a glance if a rule for one bot reaches others.
A robots.txt validator built in
It flags lines without a colon, Allow or Disallow lines that come before any User-agent, directives crawlers do not know, and noindex lines Google no longer reads, so you validate robots.txt syntax in the same test.
File status, read the way Google reads it
A missing file, a server error and no answer each mean something different to Google. The test tells you which one you got and what it means for crawling.
Sitemaps and size checked
Every Sitemap line is listed with its line number, and you are told when a file runs past 500 KiB, the point where Google stops reading.
What each robots.txt status means
| Status | What happened | What Google does |
|---|---|---|
| Found (2xx) | The file loaded | Applies the rules in it |
| Missing (4xx except 429) | There is no file | Crawls everything, so the URL counts as allowed |
| Server error (5xx or 429) | The server failed or asked crawlers to slow down | Treats the site as blocked and pauses crawling for a while |
| No answer | The file timed out, or redirected somewhere the tester does not follow | Pauses crawling for a while, so the URL counts as blocked |
On this pageHow a crawler decides which robots.txt rule applies
- How a crawler decides which robots.txt rule applies
- Robots.txt mistakes worth testing for
- What this tester does not do
How a crawler decides which robots.txt rule applies
A robots.txt file is split into groups. Each group starts with one or more User-agent lines and lists the Allow and Disallow rules for those crawlers.
- A crawler with its own group ignores the * group. If GPTBot has a group, it follows only that group. Repeat any rule from the * group that should still apply to it.
- The longest matching rule wins. Disallow: /blog loses to Allow: /blog/launch for that page, because the Allow path is longer.
- Allow wins a tie. When an Allow and a Disallow rule match with the same length, the page may be crawled.
- Wildcards count. * matches any run of characters and $ marks the end of the URL, so Disallow: /*.pdf$ blocks PDF files only.
- The query string counts too. A rule such as Disallow: /*?sort= matches the path plus everything after the question mark.
The tester applies these rules as Google documents them, then tells you which line won.
Robots.txt, robots meta tag or X-Robots-Tag?
| Method | What it controls | Where it lives | Use it when |
|---|---|---|---|
| robots.txt | Crawling: whether a bot may fetch a URL | One file at the root of each host | You want to keep crawlers out of sections, files or parameters |
| Robots meta tag | Indexing: whether a page may appear in results | In the page's HTML head | You want an HTML page kept out of search results |
| X-Robots-Tag | Indexing, for any file type | In the HTTP response header | You want a PDF, image or other file kept out of results |
A crawler has to fetch a page to see its noindex. If robots.txt blocks the page, the noindex is never read and the URL can still show up in search from links pointing to it.
On this pageRobots.txt mistakes worth testing for
- How a crawler decides which robots.txt rule applies
- Robots.txt mistakes worth testing for
- What this tester does not do
Robots.txt mistakes worth testing for
A staging Disallow: / that went live
One line blocks the whole site. The tester warns when the * group disallows everything.
Rules that reach too far
Disallow: /p also blocks /products and /pricing. Test the pages you care about, not only the folder you meant to block.
A bot group that drops your * rules
Give a crawler its own group and it stops reading the * group. The crawler table shows the effect for every bot.
Blocked CSS and JavaScript
If Google cannot fetch the files a page needs, it may not render the page as visitors see it. Test the URLs of your main stylesheet and scripts.
noindex lines in robots.txt
Google stopped reading them in 2019. The tester flags each one by line number.
Case and trailing slashes
Paths are case sensitive, and /Blog is not /blog. Disallow: /blog/ does not block /blog itself.
The wrong host
Each subdomain needs its own file. A rule on www.example.com does nothing for shop.example.com.
Who uses a robots.txt testing tool
01
SEO specialists
Find out why a page is not crawled and confirm a fix before you ask Google to recrawl it.
02
Developers
Check what a deployed file does to a real URL, including wildcards, $ anchors and Allow overrides.
03
Content and brand teams
See whether AI crawlers from ChatGPT, Claude, Perplexity and others may read your pages.
04
Site owners
After a redesign or a platform move, make sure no staging rule shipped with the site.
When to check your robots.txt
After a launch or migration
Test your homepage and a few key templates the day the new site goes live.
When Search Console says "Blocked by robots.txt"
Test that URL as Googlebot to find the line responsible.
Before and after you change AI crawler rules
Check that blocking training bots did not also block the crawlers that show your pages in AI answers.
When you audit a client or competitor site
Use it as a site robots.txt checker for any public domain, with no access to their server.
When you want to verify your sitemap line
See which sitemaps the file lists and on which line.
How free robots.txt testers compare
This table shows what each tool's own page says, checked on November 18, 2026. "Not stated" means the page does not say.
| What you get | AISEO | Tame the Bots | Scrappey | Axiolo | Pressable |
|---|---|---|---|---|---|
| Account needed to start | No | Not stated | No, but fetching from a URL needs a verification step | No | Not stated |
| Free use | 10 tests a day | Free, number not stated | Free, number not stated | "No daily limits" | Free, number not stated |
| Crawlers you can pick | Googlebot, Bingbot and 18 AI crawlers | 19 presets, 3 of them OpenAI bots, plus your own agent | 18 presets, including GPTBot, CCBot and PerplexityBot | Googlebot, Bingbot and their variants, plus your own agent | Not stated |
| File it tests | The live file | The live file or a custom one | The live file or a pasted one | The live file or a pasted one | Shows the live file |
| Deciding rule shown | Yes, with its line number | Yes, the rule; line not stated | Yes, with its line and group | Yes, the matched line | Not stated |
| Syntax problems flagged | Yes | Lines the parser does not understand | Yes, per Scrappey | Yes, with line numbers | Not stated |
| URLs per test | One | Several, one per line | One URL field | "Multiple URLs at once" | One domain |
On this pageWhat this tester does not do
- How a crawler decides which robots.txt rule applies
- Robots.txt mistakes worth testing for
- What this tester does not do
What this tester does not do
- It tests one URL per run. Run it again for the next page.
- It reads the live file only. To try rules you have not published yet, use the test panel in the robots.txt generator.
- The crawler list is fixed at 20. Image, news and ads bots and custom user agents are not on it.
- It fetches robots.txt with its own user agent, not as Googlebot. A server or firewall that treats bots differently could answer crawlers differently.
- It tells you whether a page may be crawled, not whether it is indexed.
Robots.txt tester FAQ
Using the tester
Enter any page on your site above and pick a crawler. The tester loads yourdomain.com/robots.txt, checks the URL against it, and lists the groups, sitemaps and any lines crawlers cannot read.
Google retired the robots.txt tester in Search Console in 2023 and replaced it with a robots.txt report that shows the files Google found and their errors. This tester fills the gap for any public URL: it follows Google's documented rules for groups, longest match, Allow ties and wildcards, and shows the line that decided the result.
Not here: this tester reads the live file. Build or paste your rules into the robots.txt generator and use its test panel to check paths against the draft.
Yes. You get 10 tests a day without an account.
How robots.txt works
It always sits at the root of the host, so add /robots.txt to the domain, as in example.com/robots.txt. Each subdomain has its own file. If that address returns a 404, the site has no robots.txt.
If the file answers with a 404 or another 4xx error except 429, Google treats the site as having no restrictions. A 5xx error, a 429 or no answer at all makes Google treat the site as blocked and pause crawling for a while.
Google reads the first 500 KiB and ignores the rest. The tester reads the same amount and tells you when a file goes past it.
Crawling and indexing
Not reliably. robots.txt controls crawling, not indexing, so a blocked page can still be indexed from links pointing to it. Use a noindex robots meta tag or X-Robots-Tag header on a page Google can crawl instead.
robots.txt is a request, not a lock. Most AI crawlers on the list are meant to follow it, but Perplexity says its user-triggered visits may ignore it, and Bytespider is reported not to always follow it.
More free tools for crawling and indexing
Robots.txt generator
Write a robots.txt with AI crawler rules and test paths against it.
Redirect checker
Follow every hop a URL takes and see each status code.
SEO audit
Check a page for the technical and on-page issues that hold it back.
llms.txt generator
Create an llms.txt file that points AI tools to your best pages.
Free SEO tools
Test the next URL on your list.
Run another robots.txt test, or write a cleaner file with the generator.