Developer Tools Platform
Skip to content
Categories

Robots.txt Generator

Generate a robots.txt file by specifying User-agent, Disallow, Allow, and Sitemap.

Overview

  • Generates a robots.txt file that tells search engine crawlers which paths on your site may (or may not) be crawled.
  • Simply specify Disallow/Allow paths one per line to create a correctly formatted robots.txt. You can also specify a Sitemap URL and Crawl-delay.
  • The generation happens entirely in your browser.

Usage

  1. Enter the User-agent (defaults to "*"), Disallow/Allow paths (one per line), and optionally a Sitemap URL and Crawl-delay.
  2. Click the "Generate" button.
  3. The robots.txt content is displayed. Use the "Copy" button to copy the result to your clipboard.

Example

Input

Disallow: /admin, /private
Sitemap: https://example.com/sitemap.xml

Output

User-agent: *
Disallow: /admin
Disallow: /private

Sitemap: https://example.com/sitemap.xml

Use Cases

  • Specifying paths to exclude from crawling, such as admin screens or API endpoints, when launching a new site
  • Adding a Sitemap URL to robots.txt to let search engines know where the sitemap is located
  • Reviewing or regenerating the content of an existing robots.txt from a defined set of rules

FAQ

Can I specify multiple User-agent groups separately?

This tool generates only a single User-agent group. If you need multiple groups, combine the generated results manually.

What happens if I don't specify any Disallow?

An error occurs if none of Disallow, Allow, Sitemap, or Crawl-delay is specified. To allow everything, either specify only "Allow: /" with Disallow left blank, or specify just a Sitemap.

Where should robots.txt be placed?

Place the generated content at your site's root (e.g. https://example.com/robots.txt).

Notes

  • The priority and interpretation of Disallow/Allow rules can differ between search engines. Check the documentation of major search engines after generating.
  • Nothing is sent externally — everything is processed entirely within your browser.

Related Tools

Sitemap.xml Generator

Generate a sitemap.xml file from a list of URLs.

Gitignore Generator

Select languages and tools to generate a .gitignore file.

Meta Tag Generator

Generate meta tags with OGP and Twitter Card support from a title and description.

UUID Generator

Generate multiple UUIDs (v4) at once.

Recommended