Check if crawlers can read your page.
Enter a URL and pick a crawler to test the site's live robots.txt. You see the verdict, the rule and line that decided it, and what to fix. No account needed.
- Allowed or blocked, per URL
- The deciding rule and line
- Googlebot, Bingbot and AI crawlers
- Syntax errors flagged
- Sitemaps listed
What you get
One test. The verdict and the line behind it.
The tester reads the live robots.txt on the URL's host, applies the same longest match rules Google documents, and shows every crawler's verdict side by side.
robots.txt · example.com/blog/launch
done
- file status200, 14 lines
- Googlebotallowed, no rule matches
- Bingbotallowed, no rule matches
- GPTBotblocked by Disallow: / on line 9
- line 12noindex is ignored by Google
- sitemapexample.com/sitemap.xml
Why it matters
One wrong line can hide a whole site.
robots.txt is the first file a crawler reads. A stray slash or a rule in the wrong group can keep search engines and AI assistants away from the pages you most want found.
Crawling
Keep key pages open
A page Google may not crawl cannot be read for ranking. Testing the URL shows whether a rule meant for one folder reaches it.
AI answers
Decide who reads your content
AI crawlers each follow their own group. See which ones your file lets in, so blocking training does not also keep you out of AI answers.
Errors
Catch rules crawlers skip
Misspelled directives, lines without a colon and noindex rules are ignored silently. The tester points to each one by line number.
Who it's for
Built for the people who edit robots.txt.
01
SEO specialists
Find out why a page is not crawled, and confirm a fix before you ask Google to recrawl it.
02
Developers
Check what a deployed robots.txt actually does to a URL, including wildcards, $ anchors and Allow overrides.
03
Content and brand teams
See whether ChatGPT, Claude, Perplexity and other AI crawlers may read your pages.
04
Site owners
After a redesign or a platform move, make sure a staging rule did not ship and block the site.
How it works
From URL to verdict in seconds.
01
Enter a URL
Paste a page address or just a domain, and pick the crawler to test as.
02
Read the live file
The tester fetches robots.txt from the root of the URL's host over https, the way a crawler does, and reads up to 500 KiB.
03
See the verdict
You get allowed or blocked for your crawler and every other one on the list, the deciding line highlighted in the file, and the problems found.
FAQ
Questions, answered.
It reads a site's robots.txt and tells you whether a given crawler may crawl a given URL. This one also shows the exact rule and line that decided it, so you know what to change.
Enter any page on your site and pick a crawler. The tester loads yourdomain.com/robots.txt, checks the URL against it and lists the groups, sitemaps and any lines crawlers cannot read.
Google uses the most specific rule, which is the one with the longest matching path. When an Allow and a Disallow rule match with the same length, Allow wins. The tester applies the same rules.
A crawler that has a group of its own follows only that group and skips the * group entirely. If you give GPTBot its own group, repeat any rules from the * group that should still apply to it.
If the file answers with a 404 or another 4xx error, Google treats the site as having no restrictions and crawls everything. The exception is 429, which Google treats like a server error.
When robots.txt answers with a 5xx error or does not answer at all, Google treats the whole site as blocked and pauses crawling for a while. A robots.txt that keeps failing can stop a site from being crawled.
Not reliably. robots.txt controls crawling, not indexing, so a blocked page can still be indexed from links pointing to it. Google stopped supporting noindex in robots.txt in 2019. Use a noindex robots meta tag on a page Google can crawl instead.
Google reads the first 500 KiB of the file and ignores the rest. The tester reads the same amount and tells you when a file goes past it.
The major ones, such as GPTBot, ClaudeBot and PerplexityBot, say they do. robots.txt is a request, not a lock, and some agents that fetch a page because a person asked for it may not follow it.
Yes. You can run 10 tests a day without an account. Each test reads the live file, the same one crawlers get.
Free SEO tools