Robots.txt Rules and Best Practices Explained
How Disallow, Allow, and User-agent directives work, and the mistakes that accidentally de-index a site.
robots.txt tells well-behaved crawlers which paths they may or may not request — it's a crawling instruction, not a security or indexing block. A disallowed page can still appear in search results if other sites link to it. User-agent picks which crawler a rule applies to, Disallow blocks a path, Allow carves out an exception, and rules match by most-specific path, not file order. A stray "Disallow: /" left over from staging is one of the most common causes of a site vanishing from search after a migration.
What robots.txt actually controls
robots.txt tells well-behaved crawlers which parts of a site they may or may not request — it's a crawling instruction, not a security or indexing block. A page disallowed in robots.txt can still appear in search results (usually without a description, since the crawler was never allowed to read the page content) if other sites link to it, since Google may index a URL it was never allowed to crawl based purely on link signals.
This distinction between "crawling" and "indexing" trips up a lot of people — blocking a page in robots.txt is not the reliable way to keep it out of search results. A noindex meta tag or HTTP header is the correct tool for that job, since it explicitly tells search engines not to index the page, whereas robots.txt only tells them not to fetch it.
The core directives
User-agent specifies which crawler a rule applies to (or * for all crawlers). Disallow blocks a path from being crawled. Allow carves out an exception within a disallowed path, useful when you want to block a whole directory except one specific file inside it.
Rules are matched by the most specific path, not by order in the file — a more specific Allow rule overrides a broader Disallow rule covering the same URL, regardless of which one appears first or last in the file. This trips people up when they assume rules are processed top-to-bottom like a simple script.
The mistake that de-indexes a whole site
A stray "Disallow: /" left over from a staging environment blocks crawling of the entire site — this single-line mistake is one of the most common causes of a site silently vanishing from search results after a migration or redesign. It happens because staging environments are often deliberately blocked from crawling to prevent duplicate content, and that blocking rule sometimes gets carried over accidentally when the site goes live.
Always double-check robots.txt after any site relaunch, domain migration, or CMS platform switch — a quick manual review of the file, or testing it with Search Console's robots.txt tester, takes a minute and can prevent weeks of lost search visibility.
robots.txt vs other blocking methods
| Method | Blocks crawling? | Blocks indexing? |
|---|---|---|
| robots.txt Disallow | Yes | No — can still be indexed via external links |
| Meta noindex tag | No — page is still crawled | Yes — reliably prevents indexing |
| X-Robots-Tag HTTP header | No | Yes — works for non-HTML files too |
| Password protection | Yes | Yes — crawlers can't access the content at all |
Generate a robots.txt file
- Open the Robots.txt Generator.
- Specify which paths to allow or disallow.
- Download the file and place it at your site's root.
Common pitfalls and best practices
- Using robots.txt to try to hide sensitive content. A disallowed path is still visible to anyone who looks at the robots.txt file itself, since it's a public, unauthenticated file — never rely on it for actual security.
- Blocking CSS or JavaScript files. This can prevent Google from properly rendering the page to evaluate its content and layout, which can hurt rankings — modern SEO guidance recommends allowing crawlers to fetch these resources.
- Confusing robots.txt with noindex. Disallowing a page in robots.txt doesn't reliably keep it out of search results — use a noindex tag or header for that instead.
- Not testing changes before deploying them. Search Console's robots.txt tester lets you check how a specific URL is treated before the file goes live — worth a quick check after any edit.