robots.txt tester
Last updated: 2026-10-30
Paste a robots.txt file, enter a URL and pick a crawler. The tester applies Google's documented rules (the most specific user-agent group, the longest matching rule, wildcards and the $ anchor) and shows whether the URL may be crawled and which line decides it.
A blocked URL can still appear in search results without a description if other pages link to it. To keep a page out of results, allow crawling and use a noindex tag instead.
Runs in your browser. Nothing you enter is sent anywhere.
How it works
One group per crawler
A crawler follows the group with its own user-agent name and ignores the rest; several groups with the same name are merged. Only when no group names it does it follow the * group. Version numbers and a trailing * after the name are ignored.
The longest rule wins
Among the allow and disallow rules that match the URL path, the one with the longest path decides. If an allow and a disallow rule are equally long, the allow rule wins.
Wildcards and the end anchor
* stands for any sequence of characters and $ at the end means the URL must end there. Paths are case-sensitive and include the query string.
Only four fields count
Google reads user-agent, allow, disallow and sitemap. Other lines such as crawl-delay are ignored, and the tester lists them so you know.
Based on: Google: How Google interprets the robots.txt specification
Do it for every page with GSC Magic
This checks one URL against a pasted file. With GSC Magic, your AI agent can inspect hundreds of URLs through Google's URL Inspection API and see which ones Google actually reports as blocked by robots.txt.
Inspect the URLs in our sitemap and list every one Google reports as blocked by robots.txt.
Which pages with impressions in the last 90 days are now blocked by robots.txt?
Questions
Where did the robots.txt tester in Search Console go?
Google retired the old tester and replaced it with the robots.txt report in Search Console, which shows the files Google fetched and their errors. This tool checks how a specific URL is treated by the rules.
Does blocking a page in robots.txt remove it from Google?
No. A disallowed URL can still be indexed without its content if other pages link to it. Use a noindex tag on a crawlable page to keep it out of results.
Do AI crawlers follow robots.txt?
The crawlers in the list, such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, are documented by their operators as respecting robots.txt. Each one follows its own group, or the * group when it has none.
Is the result exactly what Googlebot does?
The tester follows Google's published interpretation of the Robots Exclusion Protocol. Google also caches robots.txt for up to a day and treats server errors specially, so check the robots.txt report in Search Console after you change the file.
Guides on the blog: Blocked by robots.txt: find the rule and decide whether to unblock