Skip to content
Free robots.txt tester

Robots.txt tester: check if a URL is allowed or blocked

Enter a page and pick a crawler. The tester reads the site's live robots.txt and tells you whether that crawler may crawl the page, which rule on which line decided it, and which lines in the file crawlers will skip. You get 10 free tests a day, no account needed.

Free, no signup, 10 checks a day. Reads the live robots.txt on the URL's host.

  • Allowed or blocked, per URL
  • The deciding rule and its line
  • Googlebot, Bingbot and 18 AI crawlers
  • Unreadable lines flagged
  • Sitemaps listed
crawlers tested per URL
20
of the file read
500 KiB
free tests
10 a day
needed to test
No account

What the robots.txt checker takes and gives back

You enter
One page URL or a bare domain, up to 2,000 characters, and one crawler from the list
Crawlers
Googlebot, Bingbot and 18 AI crawlers
File it reads
/robots.txt at the root of the URL's host, over https
How much it reads
The first 500 KiB, the same amount Google reads
You get back
Allowed or blocked, the file status, how many groups and sitemaps the file has, and the rule and line that decided it
Also shown
The same URL tested for all 20 crawlers, the numbered file with the deciding line marked, and a list of problems
Price
Free, 10 tests a day, no sign-up

How to test robots.txt in three steps

Test robots.txt online from your browser. There is nothing to install and no login.

  1. 01

    Enter a URL

    Paste a page address such as example.com/blog/post, or just the domain to test the homepage. You can leave out https://.

  2. 02

    Pick a crawler

    Choose Googlebot, Bingbot or one of the AI crawlers, then select Test URL.

  3. 03

    Read the verdict

    You see allowed or blocked, the rule that decided it, the whole file with that line marked, and any lines crawlers cannot read. Fix the file, then run the test again.

What each robots.txt test shows you

  1. The rule that decided it

    The verdict names the exact Allow or Disallow line and its line number, so you know which line to change. When no rule matches, it says so.

  2. Every crawler at once

    One test of a URL fills a table of all 20 crawlers with their verdict and deciding rule. You can see at a glance if a rule for one bot reaches others.

  3. A robots.txt validator built in

    It flags lines without a colon, Allow or Disallow lines that come before any User-agent, directives crawlers do not know, and noindex lines Google no longer reads, so you validate robots.txt syntax in the same test.

  4. File status, read the way Google reads it

    A missing file, a server error and no answer each mean something different to Google. The test tells you which one you got and what it means for crawling.

  5. Sitemaps and size checked

    Every Sitemap line is listed with its line number, and you are told when a file runs past 500 KiB, the point where Google stops reading.

What each robots.txt status means

StatusWhat happenedWhat Google does
Found (2xx)The file loadedApplies the rules in it
Missing (4xx except 429)There is no fileCrawls everything, so the URL counts as allowed
Server error (5xx or 429)The server failed or asked crawlers to slow downTreats the site as blocked and pauses crawling for a while
No answerThe file timed out, or redirected somewhere the tester does not followPauses crawling for a while, so the URL counts as blocked
On this pageHow a crawler decides which robots.txt rule applies

How a crawler decides which robots.txt rule applies

A robots.txt file is split into groups. Each group starts with one or more User-agent lines and lists the Allow and Disallow rules for those crawlers.

  • A crawler with its own group ignores the * group. If GPTBot has a group, it follows only that group. Repeat any rule from the * group that should still apply to it.
  • The longest matching rule wins. Disallow: /blog loses to Allow: /blog/launch for that page, because the Allow path is longer.
  • Allow wins a tie. When an Allow and a Disallow rule match with the same length, the page may be crawled.
  • Wildcards count. * matches any run of characters and $ marks the end of the URL, so Disallow: /*.pdf$ blocks PDF files only.
  • The query string counts too. A rule such as Disallow: /*?sort= matches the path plus everything after the question mark.

The tester applies these rules as Google documents them, then tells you which line won.

Robots.txt, robots meta tag or X-Robots-Tag?

MethodWhat it controlsWhere it livesUse it when
robots.txtCrawling: whether a bot may fetch a URLOne file at the root of each hostYou want to keep crawlers out of sections, files or parameters
Robots meta tagIndexing: whether a page may appear in resultsIn the page's HTML headYou want an HTML page kept out of search results
X-Robots-TagIndexing, for any file typeIn the HTTP response headerYou want a PDF, image or other file kept out of results

A crawler has to fetch a page to see its noindex. If robots.txt blocks the page, the noindex is never read and the URL can still show up in search from links pointing to it.

On this pageRobots.txt mistakes worth testing for

Robots.txt mistakes worth testing for

  • A staging Disallow: / that went live

    One line blocks the whole site. The tester warns when the * group disallows everything.

  • Rules that reach too far

    Disallow: /p also blocks /products and /pricing. Test the pages you care about, not only the folder you meant to block.

  • A bot group that drops your * rules

    Give a crawler its own group and it stops reading the * group. The crawler table shows the effect for every bot.

  • Blocked CSS and JavaScript

    If Google cannot fetch the files a page needs, it may not render the page as visitors see it. Test the URLs of your main stylesheet and scripts.

  • noindex lines in robots.txt

    Google stopped reading them in 2019. The tester flags each one by line number.

  • Case and trailing slashes

    Paths are case sensitive, and /Blog is not /blog. Disallow: /blog/ does not block /blog itself.

  • The wrong host

    Each subdomain needs its own file. A rule on www.example.com does nothing for shop.example.com.

Who uses a robots.txt testing tool

  1. 01

    SEO specialists

    Find out why a page is not crawled and confirm a fix before you ask Google to recrawl it.

  2. 02

    Developers

    Check what a deployed file does to a real URL, including wildcards, $ anchors and Allow overrides.

  3. 03

    Content and brand teams

    See whether AI crawlers from ChatGPT, Claude, Perplexity and others may read your pages.

  4. 04

    Site owners

    After a redesign or a platform move, make sure no staging rule shipped with the site.

When to check your robots.txt

  1. After a launch or migration

    Test your homepage and a few key templates the day the new site goes live.

  2. When Search Console says "Blocked by robots.txt"

    Test that URL as Googlebot to find the line responsible.

  3. Before and after you change AI crawler rules

    Check that blocking training bots did not also block the crawlers that show your pages in AI answers.

  4. When you audit a client or competitor site

    Use it as a site robots.txt checker for any public domain, with no access to their server.

  5. When you want to verify your sitemap line

    See which sitemaps the file lists and on which line.

How free robots.txt testers compare

This table shows what each tool's own page says, checked on November 18, 2026. "Not stated" means the page does not say.

Compare with
What you getAISEOTame the BotsScrappeyAxioloPressable
Account needed to startNoNot statedNo, but fetching from a URL needs a verification stepNoNot stated
Free use10 tests a dayFree, number not statedFree, number not stated"No daily limits"Free, number not stated
Crawlers you can pickGooglebot, Bingbot and 18 AI crawlers19 presets, 3 of them OpenAI bots, plus your own agent18 presets, including GPTBot, CCBot and PerplexityBotGooglebot, Bingbot and their variants, plus your own agentNot stated
File it testsThe live fileThe live file or a custom oneThe live file or a pasted oneThe live file or a pasted oneShows the live file
Deciding rule shownYes, with its line numberYes, the rule; line not statedYes, with its line and groupYes, the matched lineNot stated
Syntax problems flaggedYesLines the parser does not understandYes, per ScrappeyYes, with line numbersNot stated
URLs per testOneSeveral, one per lineOne URL field"Multiple URLs at once"One domain
On this pageWhat this tester does not do

What this tester does not do

  • It tests one URL per run. Run it again for the next page.
  • It reads the live file only. To try rules you have not published yet, use the test panel in the robots.txt generator.
  • The crawler list is fixed at 20. Image, news and ads bots and custom user agents are not on it.
  • It fetches robots.txt with its own user agent, not as Googlebot. A server or firewall that treats bots differently could answer crawlers differently.
  • It tells you whether a page may be crawled, not whether it is indexed.

Using the tester

How robots.txt works

Crawling and indexing

Free SEO tools

Test the next URL on your list.

Run another robots.txt test, or write a cleaner file with the generator.