Robots.txt Generator
Build a clean, valid robots.txt file rule by rule: pick the bot, set Allow or Disallow, add crawl delay and sitemap. Free, no sign-up, and it runs in your browser.
Dedicated page: /en/robots-txt-generator
Quick answer
A robots.txt generator creates the robots.txt file that tells search engine crawlers which parts of your site they may crawl. Build per-bot allow and disallow rules for Googlebot and others, add a crawl delay where supported, and list your sitemap, all in valid syntax.
About this Robots.txt Generator
Build a clean, valid robots.txt file rule by rule: pick the bot, set Allow or Disallow, add crawl delay and sitemap. It is one of the SEO and Webmaster on ToolUstad, and like every tool here it runs entirely in your browser: nothing you type, paste or open is uploaded to a server, and there is no sign-up or usage limit.
What is a robots.txt file?
Robots.txt is a plain text file that lives at the root of your domain, at yoursite.com/robots.txt, and follows the Robots Exclusion Protocol. Crawlers read it before crawling and obey its rules: a User-agent line names the bot, and Disallow and Allow lines list the paths that bot may not, or may, crawl. It manages crawling, not indexing: a blocked page can still appear in results without a description if other sites link to it.
How the rules work
| Directive | Meaning | Example |
|---|---|---|
| User-agent | Which bot the group applies to | User-agent: Googlebot |
| Disallow | Paths the bot must not crawl | Disallow: /admin/ |
| Allow | An exception inside a disallowed path | Allow: /admin/help.html |
| Crawl-delay | Seconds between requests (not supported by Googlebot) | Crawl-delay: 10 |
| Sitemap | Where your XML sitemap lives | Sitemap: https://site.com/sitemap.xml |
When and why to edit robots.txt
- Protecting crawl budget: block faceted navigation, internal search results and print views.
- Keeping private areas uncrawled: admin, staging and checkout paths should not be crawled.
- Pointing bots to the sitemap: the Sitemap line helps every crawler find your pages.
- Managing AI and scraper bots: add specific user agents you want to block or allow.
Best practices
- Keep one rule group per bot and test the file with Google's robots.txt tester in Search Console.
- Use the $ anchor and * wildcard carefully: a single wrong pattern can block a whole section.
- Always include your sitemap URL and keep the file at the root of the domain, not in a folder.
- Use noindex or password protection for pages that must stay out of results entirely.
Common mistakes to avoid
- Blocking CSS and JavaScript files, which stops Google rendering your pages correctly.
- Accidentally leaving "Disallow: /" from a staging site on the live site, wiping out crawling.
- Trying to hide secret pages with robots.txt: the file is public, so it advertises their paths.
- Uploading the file with the wrong name or in a subdirectory, where crawlers never look.
Who is the robots.txt generator for?
Site owners, SEO specialists and developers launching or migrating websites. A hand written file is one typo away from blocking a whole site, while the generator builds each rule group in valid syntax and lets you download the finished file ready to upload.
A short worked example
A typical small site file looks like this: User-agent: *, then Disallow: /wp-admin/ with Allow: /wp-admin/admin-ajax.php, then a blank line and the Sitemap: line with the full sitemap URL. That shape blocks the admin area for every bot while keeping the one public endpoint WordPress needs crawlable, and it advertises the sitemap to all of them. The generator produces exactly this structure for your own paths and bots, so the only step left is uploading the file and testing it in Search Console's robots.txt tester against your most important URLs before you rely on it.
Learn: how to use the Robots.txt Generator
- Start from a preset or add your own rules, one row per rule.
- Group rules by user agent: rules for the same bot are merged into one block automatically.
- Add your sitemap URL (strongly recommended) and a crawl delay if your host is small, then copy or download robots.txt and put it in your site root.
Features
- Rule builder for any bot, with Googlebot, Bingbot and AI crawlers prelisted
- Allow and Disallow rules with automatic grouping by user agent
- Presets: allow all, block all, block admin areas, block AI training crawlers
- Crawl delay and sitemap lines included
- Live preview, copy, and a real robots.txt download
Worked example
Blocking the admin area for every bot produces:
User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml
Frequently asked questions
Where does robots.txt go?
In the root of your domain, so it is reachable at yourdomain.com/robots.txt. It must be a plain text file with exactly that name.
Does Disallow hide a page from Google completely?
No. A disallowed page can still appear in results as a link if other sites link to it, just without a description. Use a noindex tag or password protection to truly keep a page out.
Does Googlebot respect crawl delay?
No, Google ignores the crawl-delay directive; it is honoured by Bing and Yandex. Control Googlebot's pace in Search Console settings instead.
Can robots.txt block AI crawlers?
Yes for crawlers that respect it, such as GPTBot, ClaudeBot and CCBot, which is why they are in the preset list. It is a request, not a security barrier.
Does robots.txt stop a page appearing in Google?
Not reliably. Robots.txt stops crawling, but if other pages link to the blocked URL, Google can still index it with no description. To remove a page from results, use a noindex tag and allow crawling so Google can see it.
Where do I upload the robots.txt file?
In the root directory of your domain, so it loads at yoursite.com/robots.txt. On Cloudflare Pages and most static hosts it sits next to index.html. Each subdomain needs its own file.
What does Crawl-delay do and does Google respect it?
Crawl-delay asks a bot to wait a set number of seconds between requests. Bing and some smaller crawlers respect it, but Googlebot ignores it: control Google's crawl rate in Search Console settings instead.
How quickly does a new robots.txt take effect?
Crawlers usually pick up changes within a day, and Google refreshes its cached copy roughly every 24 hours. Urgent fixes can be nudged by resubmitting in Search Console. Remember the file controls future crawling only: pages already indexed stay indexed until they are recrawled or removed by other means.
Related tools
More SEO and Webmaster like this one, plus two picks from other categories. Each opens its own dedicated page.