Robots.txt Generator

It must be at the root of the host, for example https://example.com/robots.txt. A robots.txt in a subfolder is ignored, and each subdomain (shop.example.com, blog.example.com) needs its own file.

Not reliably. Disallow stops crawling, but Google can still index a blocked URL without its content if other pages link to it. To keep a page out of search results, allow crawling and add a noindex meta tag or X-Robots-Tag header instead.

Google picks the group for the most specific user agent that matches the crawler, falling back to the * group. Within that group, the rule with the longest matching path wins. If an Allow and a Disallow rule are the same length, Allow wins. The tester on this page follows the same logic, including the * and $ wildcards.

No. Google ignores Crawl-delay and adjusts its crawl rate automatically based on how your server responds. Bing and Yandex do read it. Only set it if a crawler is putting real load on your server.

Use the Block AI crawlers preset. It adds a group that disallows GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google's AI training control), CCBot (Common Crawl), PerplexityBot, Applebot-Extended, Bytespider and meta-externalagent, while keeping normal search crawlers allowed. Robots.txt is a request, not an enforcement mechanism, so crawlers that ignore it won't be stopped.

Yes. Enter a domain and the tool fetches its /robots.txt, splits it into groups and loads it into the editor. Lines the generator doesn't edit, like Host or Clean-param, are listed so you can add them back by hand if you need them.