Ad / Affiliate · pre-result
Test Your Robots.txt
A robots.txt file decides what crawlers may and may not fetch. One over-broad disallow can hide an entire section from Google, and the mistake produces no error — only a slow, unexplained loss of visibility.
Our Robots.txt Tester fetches a site's robots.txt, parses it using the same matching rules Google applies, and lets you test whether any specific path is allowed or blocked for any crawler.
No sign-up, nothing stored, results in seconds.
No need to type http:// — just the domain works. We fetch the origin's /robots.txt server-side and keep nothing. Public URLs only.
What this tool checks
File retrieval
The robots.txt file is fetched and its presence confirmed, since a missing or unreachable robots.txt changes how crawlers behave.
Rule parsing
User-agent groups, allow and disallow rules are parsed using Google's documented matching behavior, including wildcard and longest-match precedence.
Path testing
Any path is tested against the rules so the allowed-or-blocked answer for that exact URL is explicit.
Crawler targeting
Rules are evaluated per crawler, because a path can be allowed for one user-agent and blocked for another.
Deindexing risks
Over-broad disallow patterns are surfaced, since blocking a path in robots.txt can quietly remove a whole section from search.
Why it matters for SEO
Robots.txt is the first thing a crawler reads, and a single wrong line has outsized consequences. Disallowing a directory can remove hundreds of pages from the index, and because robots.txt blocks crawling rather than indexing, a blocked URL can still appear in results with no snippet, which looks broken to users. The file is small and easy to edit, which is exactly why a careless change does so much damage so quietly.
How to fix common issues
Accidentally blocking the whole site
A site-wide block is fixed by removing a "Disallow: /" rule that applies to all user-agents, since that single line hides everything from crawlers.
Blocking CSS or JavaScript
Blocked CSS or JS is fixed by allowing those resources, because Google needs to render the page and blocked assets degrade how it is understood.
Using robots.txt to deindex
Trying to deindex via robots.txt is fixed by using a noindex directive instead, since a blocked URL can still be indexed without a snippet.
Over-broad wildcard
An over-broad wildcard disallow is fixed by tightening the pattern so it matches only the URLs that genuinely should be blocked.
Wrong user-agent block
A rule blocking the wrong crawler is fixed by correcting the User-agent line so the intended bot is targeted and others are unaffected.
Frequently asked questions
What is robots.txt?
A robots.txt file is a text file at the site root that tells crawlers which paths they may and may not request.
Does robots.txt stop a page from being indexed?
Robots.txt stops crawling, not indexing. A blocked URL can still appear in results without a snippet, so noindex is the correct tool for deindexing.
Why is my page blocked when I did not intend it?
An unintended block is usually caused by an over-broad Disallow pattern or a wildcard that matches more URLs than expected.
Can different crawlers have different rules?
Robots.txt supports per-user-agent groups, so a path can be allowed for one crawler and disallowed for another within the same file.
Should I block CSS and JavaScript?
Blocking CSS and JavaScript is discouraged because Google renders pages and needs those resources to understand the content correctly.
Is this robots.txt tester free?
Our Robots.txt Tester is free, with no sign-up, no limits, and nothing stored. The file is fetched server-side, analyzed, and then discarded.
Related tools