Where do I upload the robots.txt file?

The robots.txt file must be uploaded directly to the root directory of your website domain (e.g., https://example.com/robots.txt). Placing it inside subfolders (like https://example.com/subfolder/robots.txt) has no effect and will be completely ignored by web crawlers.

Does Disallow in robots.txt remove a page from Google search?

No. Disallow prevents search engines from crawling the contents of a URL. However, if other web pages link to that URL, Google may still index the bare URL without snippets. To completely prevent indexing, use a 'noindex' meta tag instead.

Why should I avoid disallowing /wp-content/ or /assets/?

Googlebot renders web pages like a modern web browser. If you block CSS stylesheets, JavaScript files, or image directories, Google cannot render the page layout properly, which can severely damage your mobile-friendliness and search rankings.

Does Googlebot support Crawl-delay?

No. Googlebot ignores the Crawl-delay directive in robots.txt. If you need to manage Google crawl frequency, configure the crawl rate directly inside Google Search Console. Crawl-delay is respected by Bingbot, Yandex, and Baidu.

Robots.txt Generator

Generate and validate robots.txt files with customizable crawler rules and sitemap settings.

TL;DR: Robots.txt Generator is a free, browser-based tool that lets you generate and validate robots.txt files with customizable crawler rules and sitemap settings.

1. Website & Presets

2. Crawler Rules

3. XML Sitemaps

4. Live Preview

5. Validate & Export

Quick Configuration Presets

Select a standard baseline or customize your crawler directives row-by-row below:

1. Website Domain

Domain / Base URL Used to auto-populate canonical XML sitemap locations.

2. Crawler Directives

User-Agent

3. XML Sitemaps

Analyze & Import Existing robots.txt

Paste your live robots.txt below to inspect syntax, test rule conflicts, and load it into the editor:

Generated robots.txt

Valid Syntax

Critical SEO Rule: Robots.txt directs search crawlers where they can request pages. It is not an access control system and will not remove existing index entries or hide private files. Use noindex meta tags or password protection for confidential content.

Why Use This Tool?

Robots.txt Generator creates standardized robots exclusion protocol files (`robots.txt`) with custom user-agent rules (Googlebot, Bingbot, Slurp, Baiduspider), crawl-delay limits, disallow paths, and XML sitemap references.

Getting Started with the Robots.txt Generator

Choose default access policy: Allow all crawlers or Restrict access.
Add Disallow rules for private directories (e.g., `/admin/`, `/cart/`, `/temp/`).
Specify custom crawler rules (e.g. block AI scrapers or aggressive bots).
Add your XML Sitemap URL and download your finalized `robots.txt` file.

Pro Tips & Best Practices

Place in Web Root: The file must reside strictly in the top-level root directory (`https://example.com/robots.txt`).
Always Reference Sitemaps: Include `Sitemap: https://example.com/sitemap.xml` at the bottom of your robots.txt.
Case-Sensitive Paths: Remember that robots.txt path matching is case-sensitive on Unix/Linux servers.

Related Guides & Tutorials

What Is Schema Markup Json Ld — Learn in-depth concepts, best practices, and expert tips.
Robots Txt Rules Best Practices — Learn in-depth concepts, best practices, and expert tips.

100% Client-Side Privacy Guarantee

All processing runs locally inside your browser using JavaScript and HTML5 APIs. Your data, files, and inputs are never uploaded to any remote server. Complete privacy by design.

Frequently Asked Questions

: The robots.txt file must be uploaded directly to the root directory of your website domain (e.g., https://example.com/robots.txt). Placing it inside subfolders (like https://example.com/subfolder/robots.txt) has no effect and will be completely ignored by web crawlers.
: No. Disallow prevents search engines from crawling the contents of a URL. However, if other web pages link to that URL, Google may still index the bare URL without snippets. To completely prevent indexing, use a 'noindex' meta tag instead.
: Googlebot renders web pages like a modern web browser. If you block CSS stylesheets, JavaScript files, or image directories, Google cannot render the page layout properly, which can severely damage your mobile-friendliness and search rankings.
: No. Googlebot ignores the Crawl-delay directive in robots.txt. If you need to manage Google crawl frequency, configure the crawl rate directly inside Google Search Console. Crawl-delay is respected by Bingbot, Yandex, and Baidu.

Cite This Tool

Referencing this tool in a paper, article, or bibliography? Copy a ready-made citation below.

NexLove.org. (2026). Robots.txt Generator [Software]. https://nexlove.org/tools/robots-txt-generator.html

“Robots.txt Generator.” NexLove.org, 2026, nexlove.org/tools/robots-txt-generator.html.

NexLove.org (2026) Robots.txt Generator. Available at: https://nexlove.org/tools/robots-txt-generator.html (Accessed: 2026).

@misc{nexlove_robots_txt_generator,
  title = {Robots.txt Generator},
  author = {{NexLove.org}},
  year = {2026},
  url = {https://nexlove.org/tools/robots-txt-generator.html}
}

Embed This Tool

Add the live Robots.txt Generator tool to your own website with this snippet — it loads the real, working tool in an iframe, not a static screenshot.

<iframe src="https://nexlove.org/embed/robots-txt-generator.html" width="100%" height="600" style="border:1px solid #e2e8f0;border-radius:12px" title="Robots.txt Generator — NexLove.org" loading="lazy"></iframe>

Related SEO Tools

Twitter Card Generator

Robots.txt Validator

Browse all SEO Tools