Glossary
robots.txt
robots.txt is a plain text file at the root of a website that tells crawlers, such as the one Google uses, which addresses on the site they may fetch. Its job is managing crawler traffic. Keeping a page out of search results takes a different tool.
What the file is for
The file is a set of rules. Each rule names a crawler, or all of them, and lists the paths it should leave alone. Google says the main purpose is to manage crawler traffic so your server is not overloaded with requests, and that robots.txt is not a way to keep a page out of Google (Google Search Central).
A page blocked in robots.txt can still be indexed if other pages link to it, because Google can list an address without reading what is on it. To keep a page out of results, Google points to a noindex rule or a password. Google can obey noindex only on a page it is allowed to fetch, because the rule arrives with the page itself (Google on noindex).
The line that hides a whole site
One line is enough: Disallow: /, placed in the group that starts with User-agent: *. It tells every crawler that follows the file to fetch nothing at all. The usual story is a redesign: someone blocked all crawlers while the new site was being built, and the block survived the launch. Nothing in a browser looks wrong.
The free scan reads your robots.txt first. If the file shuts out every crawler, ours included, the scan stops there and says so, and it tells you when Google is shut out too. A block aimed at Googlebot alone, or at pages your homepage menu links to, is T1, a high-severity finding. Either way, the fix is to delete the line or narrow it to the private areas it was meant for.
AI search crawlers read it too
The same file speaks to the crawlers that AI assistants use to find pages for their answers. A rule for every crawler, under User-agent: *, covers them too unless the file gives them rules of their own. The scan checks for this separately as A1. Blocking crawlers that only gather AI training data is your call, and the scan notes it without counting it against you.
When your site builder writes it for you
Google notes that site builders often manage robots.txt themselves (Google Search Central). If yours does, you may have no file to edit. Look in the site settings for an option that hides the site from search engines, and make sure it is switched off now that the site is live.
What the free scan checks
Questions
Should I block anything in robots.txt?
Only when you have a reason. An admin or checkout path is the usual candidate, to save crawlers a wasted trip. Remember the file is public and a block is no lock: anyone can read the paths you list. Put a password on anything private.
How do I see the robots.txt on my own site?
Type your domain followed by /robots.txt into a browser, as in example.com/robots.txt. If rules appear, read each Disallow line with the User-agent line above it. A page-not-found error means you have no file, so nothing is blocked. A server error is worse: crawlers can treat it as a request to stay out, so ask whoever hosts your site to fix it.
Where this fits in our work
We handle this as part of Technical SEO: The crawl, index, speed and structured data problems that keep good pages from being found, fixed and shown with evidence.
Related reading
Guides
- What should a local SEO audit check? The checks a local SEO audit should cover, which ones our free scan runs, and what only access to your accounts can show.
Posts that use this term
- Can I see what ChatGPT or Google's AI says about my business? No assistant shows you the words it used about your business, so you ask. A list of questions to put to each one, and what to record each round.
Related terms
- XML sitemap: A list of the pages you want search engines to find, and when a small business site needs one.
- noindex: The rule that keeps a page out of search results, and how it ends up on pages that should be found.
- Core Web Vitals: Google's three measures of loading, responsiveness and visual stability, and the thresholds for a good score.
- Largest Contentful Paint (LCP): How long the biggest thing on the screen takes to appear, and what usually slows it down.
Sources
- Google Search Central: Introduction to robots.txt, updated December 10, 2025. Checked September 26, 2026.
- Google Search Central: Block Search indexing with noindex, updated December 10, 2025. Checked September 26, 2026.