Most site owners have read their robots.txt file exactly once, if ever. It is a handful of lines, it lives at a URL nobody visits, and it does not appear anywhere in the admin interface of the thing that generated it.
It is also one of the few files that can remove an entire site from search results without breaking a single page.
What the file actually does
robots.txt tells crawlers where they may go. It is a request, not a lock. Well-behaved crawlers honour it. It does not secure anything, it does not hide anything from people, and it does not remove pages that are already indexed.
That last point causes most of the confusion. Blocking a page in robots.txt does not delete it from search. It stops the crawler from reading the page, which means the crawler can no longer see the content, the canonical tag, or a noindex instruction. A blocked page can sit in the index indefinitely with no description under it.
If the goal is removal, noindex is the tool, and the page has to stay crawlable for the crawler to see it.
The rule that does the damage
The single most expensive line is this one:
``` User-agent: * Disallow: / ```
That blocks everything. It is the correct configuration for a staging site, which is exactly why it ends up on production. A staging environment is cloned, promoted, or restored from backup, and the file comes with it.
The failure has a specific shape: nothing looks wrong. Pages load. Forms work. Speed is fine. Traffic declines over days or weeks as the crawler stops refreshing what it already has, and by the time impressions drop far enough to notice, the cause is a month behind you.
The smaller rules that still cost something
Not every mistake is total. A few common ones:
A Disallow on a directory that has since become a real content path. A blog moves from /news/ to /blog/, but the old rule blocking /blog/ from an earlier experiment is still there.
A blocked assets directory. If CSS or JavaScript is unreachable, the crawler renders a version of the page that no visitor sees. Layout and content checks are performed against that broken render.
A sitemap reference pointing at a URL that no longer exists, so the file advertises a 404.
Wildcard rules copied from a tutorial that match more than intended. Disallow: /*? looks like it blocks tracking parameters and also blocks every filtered or paginated URL on the site.
Why it drifts
robots.txt is rarely edited on purpose. It changes as a side effect.
A platform migration writes a default file. A plugin or hosting toggle labelled "discourage search engines" rewrites it. A CDN or edge rule starts serving a different file than the origin. A framework generates it at build time from a config value that someone changed for an unrelated reason. A restored backup brings back a version from before the last fix.
In each case the person making the change is not thinking about crawlers, and no deploy step fails.
How to check it properly
Fetching the file in a browser is a start, but it is not the check.
Request it the way a crawler would, from outside your network, and confirm three things: it returns 200, the content type is text/plain, and the body is what you expect. A robots.txt that returns 404 is usually fine, because no file means no restrictions. A robots.txt that returns your HTML error page is not fine, and some crawlers will treat an unparseable file as a reason to back off.
Then check the file against a page you care about. Take your highest-value URL and confirm no rule matches it. Search Console will tell you directly for a Google crawler, and it will also show you the last time it fetched the file.
Then check that the sitemap line resolves.
The important part is that this is not a one-time task. The file did not break on the day you launched. It breaks on the day someone restores a backup, and the only way to catch that is to look again on a schedule.
What monitoring should tell you
A monitor that only reports uptime will show green through all of this. The page is up. The file is served. The status code is 200.
What matters is whether the content changed and whether it now blocks something it did not block yesterday. That is a comparison against a known-good state, not a liveness check, and it is the difference between knowing your site is reachable and knowing your site is still findable.
If you want that comparison running on your own site without setting it up yourself, start here.