User-agent group 1
Choose a crawler and define its access rules.
Development
Robots.txt Generator
Create, import, validate, copy and download robots.txt files with a visual rule builder.
Robots.txt builder
Build crawler rules visually, import an existing file, validate the result and download a production-ready robots.txt file.
Templates
Start from a common configuration and customize it for your site.
General
CMS
Framework
E-commerce
Site type
Crawler control
Configuration
Configure crawler groups, global directives and sitemap URLs.
Sitemaps
Add absolute URLs for sitemap files or sitemap indexes.
No sitemap URLs added.
Preferred host
Optional directive with limited crawler support.
File comments
One comment per line. The generator adds the # prefix.
About this tool
What is Robots.txt Generator?
Build a standards-friendly robots.txt file visually without writing every directive by hand. Add crawler groups, Allow and Disallow rules, sitemap URLs, crawl delays and optional host directives.
Features
Built for fast developer workflows
- Visual user-agent and rule builder
- Import and parse existing robots.txt files
- Templates for common frameworks, CMS platforms and crawler policies
- Real-time preview and validation
- Copy or download the generated robots.txt file
Frequently asked questions
Questions about Robots.txt Generator
What is a robots.txt file?
A robots.txt file gives instructions to compliant web crawlers about which paths they may request. It is not an access-control or security mechanism.
Where should robots.txt be placed?
Publish it at the root of the relevant origin, for example https://example.com/robots.txt.
Does Disallow protect private content?
No. Disallow is only a crawler instruction. Protect private content with authentication and server-side authorization.
Can I include multiple sitemap URLs?
Yes. A robots.txt file may include multiple Sitemap directives with absolute URLs.
Related tools