A one-line mistake that has taken down entire sites
robots.txt is one of the smallest files on a website and one of the highest-leverage places to make a catastrophic mistake. A single stray line, Disallow: /, left behind from a staging environment or added by a developer temporarily blocking crawlers during a migration, blocks Googlebot from the entire site. Unlike most SEO mistakes, this one is not gradual: it can take effect on the very next crawl, and because robots.txt directives are a request Google actually honors (unlike advisory signals it can override), there is no ambiguity or grace period, Google simply stops requesting pages it is told not to request.
The dangerous part is how invisible this is from a normal site visit. A visitor loads the homepage fine, every internal link works, nothing looks broken. robots.txt is not rendered anywhere in the UI; the only way to catch a bad rule is to actually read the file or run it through a parser that checks it the way a crawler would, which is what this tool does.
How the Robots Exclusion Protocol actually resolves conflicting rules
robots.txt was an informal, inconsistently implemented convention for over 25 years before it became an official internet standard. In September 2022, the IETF published RFC 9309, formalizing the Robots Exclusion Protocol with Google, Bing, and other major crawler operators as co-authors, which finally gave the format a precise, documented conflict-resolution algorithm instead of "whatever each crawler happens to do." The rule, in short: for a given crawler, find the most specific matching User-agent group (an exact name match beats a "*" wildcard group); within that group, find the longest matching path pattern across all Allow and Disallow rules; the longest match wins, and an exact-length tie goes to Allow. This checker implements that exact algorithm rather than a simplified approximation, which is why its verdicts match what Googlebot itself would actually do, not just a rough guess.
Disallow is not noindex, and confusing them backfires
The single most consequential misunderstanding in this space is treating Disallow as a way to remove a page from search results. It is not. Disallow only withholds permission to crawl (fetch and read) a URL; it says nothing about whether Google may still show that URL in search results. If any other page anywhere links to a disallowed URL, Google can and often does index the bare URL anyway, displaying it with no title and no snippet since it was never allowed to read the page to generate either. The counterintuitive fix: to genuinely keep a page out of the index, the page must NOT be blocked by robots.txt, so Googlebot can crawl it and see a noindex meta tag or X-Robots-Tag header on the page itself. Blocking a page you also want deindexed with robots.txt actively prevents the deindexing signal from ever being seen.
What this tool actually checks
Beyond the headline "can Googlebot reach my homepage" check, this tool parses every User-agent group, every Allow/Disallow rule, and every Sitemap directive in the file, then runs a set of diagnostic passes: a missing Sitemap declaration (not fatal, but a free, easy win for crawler discovery), Crawl-delay directives that Googlebot silently ignores, contradictory Allow/Disallow pairs targeting the identical path, malformed lines that don't match any recognized directive syntax, and the specific WordPress pattern of blocking /wp-admin/ without the commonly recommended /wp-admin/admin-ajax.php exception. None of these individually will always break a site the way a blanket Disallow: / can, but each represents either wasted crawl budget, a missed opportunity, or a signal that the file was edited carelessly and deserves a closer read.
Paste mode versus live-fetch mode
Live-fetch mode is for auditing a site that is already deployed: enter a domain and the tool retrieves {origin}/robots.txt directly over HTTPS, the same way a real crawler would. Paste mode exists for the case that matters most for prevention rather than cleanup: validating a new robots.txt before it goes live, during a migration, a staging-to-production promotion, or a CDN configuration change, so a catastrophic rule never reaches production in the first place. Both modes run through the identical parsing and diagnosis engine, so a clean result in paste mode is a genuine guarantee of what the live version will do once deployed, not a rough approximation.
Wildcards, end-anchors, and other syntax that trips people up
robots.txt supports two special characters beyond literal path matching: * matches any sequence of characters (including none), and $ anchors a rule to the end of the URL. The difference between Disallow: /private and Disallow: /private$ is easy to miss and consequential: the first blocks every URL beginning with /private, including /private-notes and /private/anything, while the second blocks only the exact path /private and nothing else. A site intending to block one specific page but accidentally omitting the trailing $ can end up blocking an entire, unrelated section that happens to share the same path prefix, a bug this checker's path-crawlability table is specifically built to surface for the four paths it tests.
Case sensitivity is easy to get wrong
Paths in robots.txt are matched case-sensitively. Disallow: /Private and a real URL at /private (lowercase) do not match at all, meaning the rule silently does nothing for that URL. This becomes a real problem on sites with inconsistent URL casing across their own codebase, a legacy CMS section using capitalized paths while newer pages use lowercase, since a robots.txt written against one casing convention provides zero protection for the other. If a path genuinely needs to be blocked regardless of casing, both variants need their own explicit rule; there is no case-insensitive matching mode in the specification.
Multiple User-agent lines under one rule set
A single group of Allow/Disallow rules can apply to several crawlers at once by stacking consecutive User-agent: lines before the rules, for example naming both Googlebot and Bingbot back to back, followed by one shared set of Allow/Disallow lines that applies to both. The moment a rule line (Allow, Disallow, or Crawl-delay) appears, the group is considered closed; any further User-agent: line after that starts a brand-new group instead of adding to the previous one. Misunderstanding this boundary is a common cause of a rule silently applying to fewer crawlers than intended, since a rule written after an unrelated crawler's block was accidentally closed ends up in its own orphaned group instead of the one the file's author meant to extend.
What a 404 versus a 500 on robots.txt actually means for your crawl
Google's crawling documentation treats a missing robots.txt very differently from a broken one. A clean 404 response for /robots.txt is interpreted as "no crawling restrictions exist at all" — Google proceeds to crawl the entire site as if the file simply granted unrestricted access, which is a legitimate and common configuration for smaller sites that never bothered to add one. A 5xx server error, on the other hand, is treated as an unknown state rather than an all-clear signal: Google's documented behavior is to assume access might be restricted and hold back crawling of the site until it can fetch a real robots.txt response, since a server error might be temporarily masking rules the site actually wants enforced. In practice this means a robots.txt endpoint that occasionally times out or returns a 500 under load, even if every other page on the site is healthy, can throttle your entire site's crawl rate without any other visible symptom. This checker's fetch mode surfaces the literal HTTP status it received so a flaky robots.txt endpoint is caught explicitly rather than silently treated the same as a clean 404.
Sitemap directives belong in robots.txt even if you already submitted one in Search Console
Submitting a sitemap through Google Search Console and declaring it in robots.txt are not redundant, they serve different crawlers and different discovery paths. The Sitemap: directive in robots.txt is readable by any crawler that fetches the file at all, including ones that were never configured through a webmaster console, Bing, smaller search engines, and third-party SEO crawlers doing their own site audits. It is also the only sitemap-discovery mechanism that works without any prior manual setup step, which matters for a newly launched site that has not yet been verified in Search Console. Declaring the sitemap in both places costs nothing and closes a discovery gap that costs real, if usually small, crawl efficiency on any site that skips it.
Non-standard directives you may see in the wild
Two directives show up in real robots.txt files despite never being part of the official RFC 9309 standard: Host, historically used by Yandex to declare a preferred mirror domain, and Clean-param, also Yandex-specific, used to tell that crawler which URL query parameters can be ignored for deduplication purposes. Neither is read by Googlebot or Bingbot. This checker parses both without flagging them as errors, since they are valid syntax for the crawlers that do support them, but they contribute nothing to Google or Bing crawl behavior and should not be relied on as a substitute for a real canonical tag or a genuine Google-specific directive.
Frequently asked questions
What is robots.txt and where does it need to live?
robots.txt is a plain-text file that must sit at the root of a domain, https://yoursite.com/robots.txt, nowhere else, or crawlers will not find it. It implements the Robots Exclusion Protocol, formally standardized as RFC 9309 in September 2022, which tells well-behaved crawlers which parts of a site they may request. It is a request, not an enforcement mechanism: it cannot prevent a page from being crawled by a bot that ignores it, and it cannot remove a page already in Google's index (that requires a noindex meta tag or header instead).
Does Disallow in robots.txt remove a page from Google's index?
No, and this is the single most common robots.txt misunderstanding. Disallow only tells Googlebot not to crawl (fetch) a URL. If other pages already link to that URL, Google can still index the URL itself, showing it in search results with no title or snippet, "No information is available for this page", because Google indexed the URL reference without ever being allowed to read the page content. To actually keep a page out of the index, use a noindex meta tag or X-Robots-Tag header, and make sure the page is NOT blocked by robots.txt, since Googlebot has to be able to crawl the page to see the noindex instruction in the first place.
Why does this tool check /, /sitemap.xml, /blog, and /product specifically?
These four represent the most common accidental self-inflicted wounds: a blanket block on the homepage or entire site (usually left over from a staging-site robots.txt that never got updated for production), a blocked sitemap file, and blocked category paths that are frequently swept up by an overly broad Disallow rule aimed at something else, like an /admin or /wp-admin path that also happens to match /blog or /product as a prefix.
Does Google support Crawl-delay?
No. Google has stated directly that Googlebot does not support the non-standard Crawl-delay directive and ignores it entirely; crawl rate for Google is instead managed through Search Console's crawl-rate settings or, more commonly today, left to Google's own automatic rate-limiting. Bing and Yandex do honor Crawl-delay, so it is not useless, just ineffective specifically for Googlebot, which is why this checker flags it as informational rather than an error.
What happens if robots.txt returns a 404 or a server error?
Google's documentation draws a sharp distinction here. A 404 (file not found) is treated as "no restrictions exist" — Google crawls as if there were no robots.txt at all, which is a valid, intentional configuration many small sites run. A 5xx server error, however, is treated cautiously: Google will typically pause crawling of the site entirely for a period rather than assume unrestricted access, on the theory that the server error might be masking a real robots.txt the site intended to serve. A robots.txt that intermittently 500s is a crawl-budget risk even if the site otherwise works fine for visitors.
Can I have different rules for Googlebot versus other bots?
Yes — that is the entire purpose of the User-agent directive. A file can have one group of rules under "User-agent: *" that applies to every crawler by default, and a more specific group under "User-agent: Googlebot" that overrides it for Google specifically. When a crawler reads the file, it uses the most specific group that names it (or matches its name as a prefix) and ignores the generic "*" group entirely for that crawler, it does not merge the two.
My robots.txt has both an Allow and a Disallow for the same path. Which wins?
Per Google's documented algorithm, the most specific rule wins, meaning the longest matching path pattern takes precedence, since a longer, more specific path is assumed to reflect deliberate intent over a shorter, broader one. If two rules end up exactly the same length after matching, Allow wins the tie. This checker flags any case where an Allow and Disallow target the exact identical path string, since that specific case is almost always an accidental leftover rule rather than an intentional override.
Need your full technical SEO checked, not just one file?
Pair this with the Schema Markup Generator and Cookie Consent Checker below for a fuller pre-launch technical audit, all free, all in one place.
Primary sources: RFC 9309 — Robots Exclusion Protocol; Google Search Central — How Google interprets robots.txt; Google Search Central — Create and submit a robots.txt file; Google Search Central — Robots meta tag, data-nosnippet, and X-Robots-Tag.