Does robots.txt keep a page out of search results?

Not reliably. robots.txt controls crawling, not indexing: a disallowed page that other sites link to can still appear in results without a snippet. To keep a page out of the index, allow crawling and add or an X-Robots-Tag header. robots.txt is also public and voluntary, so never rely on it to hide sensitive paths.

Will blocking Google-Extended hurt my Google rankings?

No. Google-Extended is only a control token that decides whether your content may be used to train Gemini and other generative models; Googlebot’s normal crawling and ranking are unaffected. Likewise, blocking training crawlers such as GPTBot, ClaudeBot or CCBot doesn’t affect classic search engines — but blocking AI search crawlers like OAI-SearchBot or PerplexityBot may keep your site out of those AI search answers.

Which wins when Allow and Disallow conflict?

Under Google’s rules and RFC 9309, among all rules that match the URL the one with the longest (most specific) path wins, and Allow wins when lengths are equal. Order in the file doesn’t matter. For example, if Allow: /page and Disallow: /*.htm both match /page.htm, the Disallow rule is longer, so the URL is blocked. The tester shows exactly which line decided each result.

Do changefreq and priority still matter in sitemaps?

Google has said it ignores changefreq and priority and only uses lastmod when it’s accurate; other engines such as Bing may still read them. So you can leave them out, but only update lastmod when the content really changes. A single sitemap may hold at most 50,000 URLs and 50MB uncompressed; larger sites must split into several files listed in a sitemap index, which this tool does automatically.

🤖

🤖robots.txt Generator & Tester + XML Sitemap Generator

robots.txt and sitemap.xml are the basics of technical SEO: one tells crawlers what they may fetch, the other tells search engines which pages exist. This tool covers four jobs. robots.txt generator: write Allow / Disallow rules and Crawl-delay per User-agent group and add Sitemap lines; 16 AI crawlers — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot, CCBot, Bytespider, meta-externalagent and more — can each be allowed or blocked, or you can block every training crawler in one click. robots.txt tester: paste any robots.txt and a list of URLs, pick a crawler, and see whether each URL is allowed under RFC 9309 and Google’s implementation, along with the exact line that decided it — * wildcards and $ anchors, longest match wins, Allow wins ties, groups with the same name are merged, and Googlebot-Image falls back to Googlebot. Sitemap generator: one URL per line with optional lastmod, changefreq, priority, image and video fields; reciprocal hreflang links and x-default are added automatically for language paths such as /en/ and /de/, files are split at 50,000 URLs with a sitemap index, and everything can be downloaded gzipped. Sitemap validator: checks the namespace, URL format, dates, value ranges, duplicates, size and count limits, and whether hreflang links are reciprocal.

Free · No sign-up · Runs in your browserLast updated: 2026-10-26

Loading tool...

How to use robots.txt & Sitemap Generator

1On the robots.txt tab, set up groups, AI crawler switches and Sitemap URLs, then copy or download
2On the tester tab, paste a robots.txt and URLs and pick a crawler to see allowed or blocked and the matching rule
3Paste a URL list on the sitemap tab to build sitemap.xml, or paste an existing file on the validator tab to check it

FAQ

Does robots.txt keep a page out of search results?: Not reliably. robots.txt controls crawling, not indexing: a disallowed page that other sites link to can still appear in results without a snippet. To keep a page out of the index, allow crawling and add <meta name="robots" content="noindex"> or an X-Robots-Tag header. robots.txt is also public and voluntary, so never rely on it to hide sensitive paths.
Will blocking Google-Extended hurt my Google rankings?: No. Google-Extended is only a control token that decides whether your content may be used to train Gemini and other generative models; Googlebot’s normal crawling and ranking are unaffected. Likewise, blocking training crawlers such as GPTBot, ClaudeBot or CCBot doesn’t affect classic search engines — but blocking AI search crawlers like OAI-SearchBot or PerplexityBot may keep your site out of those AI search answers.
Which wins when Allow and Disallow conflict?: Under Google’s rules and RFC 9309, among all rules that match the URL the one with the longest (most specific) path wins, and Allow wins when lengths are equal. Order in the file doesn’t matter. For example, if Allow: /page and Disallow: /*.htm both match /page.htm, the Disallow rule is longer, so the URL is blocked. The tester shows exactly which line decided each result.
Do changefreq and priority still matter in sitemaps?: Google has said it ignores changefreq and priority and only uses lastmod when it’s accurate; other engines such as Bing may still read them. So you can leave them out, but only update lastmod when the content really changes. A single sitemap may hold at most 50,000 URLs and 50MB uncompressed; larger sites must split into several files listed in a sitemap index, which this tool does automatically.