Log in Sign up free

Robots.txt Generator

Build a valid robots.txt with sitemaps, crawl rules and AI-bot controls.

Runs instantly in your browser — results update as you type.

What is robots.txt?

robots.txt is a plain-text file at the root of a website (example.com/robots.txt) that tells crawlers which paths they may or may not request. It manages crawling — not indexing — and can also point bots to your XML sitemap.

robots.txt syntax in one minute

User-agent: *          # rules for all crawlers
Disallow: /admin/      # do not crawl this folder
Allow: /admin/help     # except this path
Sitemap: https://example.com/sitemap.xml

Each group starts with one or more User-agent lines followed by Disallow and Allow rules. A crawler uses the most specific group that matches its name. * matches any characters and $ marks the end of a URL (Disallow: /*.pdf$).

What to block — and what never to block

  • Usually block: admin areas, internal search results (/search, /?s=), cart and checkout steps, staging folders, infinite filter combinations.
  • Never block: CSS and JavaScript files (Google needs them to render pages), or pages you want indexed.
  • Do not use robots.txt to hide pages from Google. A disallowed URL can still be indexed from links, just without a description. Use noindex instead — and do not disallow that page, or Google will never see the noindex.

Blocking AI crawlers

The AI option adds rules for crawlers that collect training data: GPTBot and ChatGPT-User (OpenAI), CCBot (Common Crawl), Google-Extended (Gemini training — does not affect Google Search), ClaudeBot and anthropic-ai, PerplexityBot, Bytespider, Applebot-Extended and Meta's crawler. Blocking them does not change your Google rankings, but it may reduce how often AI assistants cite your content — a trade-off for GEO. Reputable AI companies honour these rules.

Crawl-delay

Bing and Yandex respect Crawl-delay; Google ignores it (manage Googlebot's rate in Search Console). Only use it if a crawler is overloading your server.

Installing and testing

  1. Download the file and upload it to the root of your domain so it is reachable at /robots.txt. Each subdomain needs its own file.
  2. Test any path with the Robots.txt Checker.
  3. Make sure your sitemap URL is correct with the Sitemap Checker.

Frequently asked questions

Where do I put robots.txt?

In the root of your domain so it loads at https://yourdomain.com/robots.txt. Each subdomain needs its own robots.txt file.

Does robots.txt stop a page from being indexed?

No. It stops crawling. A blocked URL can still be indexed from links. Use a noindex meta tag on a crawlable page to keep it out of results.

Will blocking GPTBot hurt my Google rankings?

No. GPTBot is OpenAI's crawler. Blocking it has no effect on Google Search, though AI assistants may cite your site less.

Does Google respect Crawl-delay?

No. Google ignores Crawl-delay. Bing and Yandex respect it.

Should I block CSS and JavaScript?

No. Google needs CSS and JavaScript to render your pages correctly; blocking them can hurt rankings.