# robots.txt Generator & Tester + XML Sitemap Generator > Build robots.txt (block GPTBot, ClaudeBot and other AI bots in one click), test URLs with Google’s rules, and generate or validate sitemap.xml. - URL: https://qilingtools.com/en/robots-sitemap - Category: Developer Tools (https://qilingtools.com/en/dev) - Tags: [Developer](https://qilingtools.com/en/tags/dev), [Frontend](https://qilingtools.com/en/tags/frontend), [Network & HTTP](https://qilingtools.com/en/tags/network) - Last updated: 2026-10-26 - Price: Free, no sign-up - Privacy: Runs locally in the browser; files are never uploaded robots.txt and sitemap.xml are the basics of technical SEO: one tells crawlers what they may fetch, the other tells search engines which pages exist. This tool covers four jobs. robots.txt generator: write Allow / Disallow rules and Crawl-delay per User-agent group and add Sitemap lines; 16 AI crawlers — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot, CCBot, Bytespider, meta-externalagent and more — can each be allowed or blocked, or you can block every training crawler in one click. robots.txt tester: paste any robots.txt and a list of URLs, pick a crawler, and see whether each URL is allowed under RFC 9309 and Google’s implementation, along with the exact line that decided it — * wildcards and $ anchors, longest match wins, Allow wins ties, groups with the same name are merged, and Googlebot-Image falls back to Googlebot. Sitemap generator: one URL per line with optional lastmod, changefreq, priority, image and video fields; reciprocal hreflang links and x-default are added automatically for language paths such as /en/ and /de/, files are split at 50,000 URLs with a sitemap index, and everything can be downloaded gzipped. Sitemap validator: checks the namespace, URL format, dates, value ranges, duplicates, size and count limits, and whether hreflang links are reciprocal. ## How to use 1. On the robots.txt tab, set up groups, AI crawler switches and Sitemap URLs, then copy or download 2. On the tester tab, paste a robots.txt and URLs and pick a crawler to see allowed or blocked and the matching rule 3. Paste a URL list on the sitemap tab to build sitemap.xml, or paste an existing file on the validator tab to check it ## Features - Visual editor for User-agent groups, Allow / Disallow and Crawl-delay - Allow or block 16 AI crawlers individually or all training bots at once - URL tester using Google’s longest-match rules, showing the matching line - * wildcards, $ anchors, merged groups and crawler fallback - Sitemaps with lastmod, changefreq, priority, images and videos - Automatic reciprocal hreflang and x-default for language paths - Auto-split at 50,000 URLs with a sitemap index and gzip download - Sitemap validation: format, values, duplicates, limits and hreflang ## FAQ ### Does robots.txt keep a page out of search results? Not reliably. robots.txt controls crawling, not indexing: a disallowed page that other sites link to can still appear in results without a snippet. To keep a page out of the index, allow crawling and add or an X-Robots-Tag header. robots.txt is also public and voluntary, so never rely on it to hide sensitive paths. ### Will blocking Google-Extended hurt my Google rankings? No. Google-Extended is only a control token that decides whether your content may be used to train Gemini and other generative models; Googlebot’s normal crawling and ranking are unaffected. Likewise, blocking training crawlers such as GPTBot, ClaudeBot or CCBot doesn’t affect classic search engines — but blocking AI search crawlers like OAI-SearchBot or PerplexityBot may keep your site out of those AI search answers. ### Which wins when Allow and Disallow conflict? Under Google’s rules and RFC 9309, among all rules that match the URL the one with the longest (most specific) path wins, and Allow wins when lengths are equal. Order in the file doesn’t matter. For example, if Allow: /page and Disallow: /*.htm both match /page.htm, the Disallow rule is longer, so the URL is blocked. The tester shows exactly which line decided each result. ### Do changefreq and priority still matter in sitemaps? Google has said it ignores changefreq and priority and only uses lastmod when it’s accurate; other engines such as Bing may still read them. So you can leave them out, but only update lastmod when the content really changes. A single sitemap may hold at most 50,000 URLs and 50MB uncompressed; larger sites must split into several files listed in a sitemap index, which this tool does automatically. ## Related tools - [JSON Formatter](https://qilingtools.com/en/json): Free online JSON formatter: beautify or minify, validate with the exact line and column of any error, browse a collapsible tree, sort keys and escape strings. - [Base64 Encode/Decode](https://qilingtools.com/en/base64): Free online Base64 encoder and decoder: UTF-8 safe, optional URL-safe variant, convert text to Base64 and back in real time. Processed locally, never uploaded. - [URL Encode/Decode](https://qilingtools.com/en/url-encode): Free online URL encoder and decoder with encodeURIComponent and encodeURI modes. Escapes Chinese, spaces and special characters. Real-time and local, no upload. - [JWT Decoder](https://qilingtools.com/en/jwt): Free online JWT decoder: decode the header and payload of a JSON Web Token, show expiry and issued-at times, all locally without uploading the token. - [Timestamp Converter](https://qilingtools.com/en/timestamp): Free online Unix timestamp converter: convert timestamps and dates, auto-detect seconds or milliseconds, and show local, UTC, ISO 8601 and relative time. - [Hash / Checksum](https://qilingtools.com/en/hash): Free online hash calculator: SHA-1, SHA-256, SHA-384 and SHA-512 of text or any file through the browser crypto API. Files never leave your device. - [UUID Generator](https://qilingtools.com/en/uuid): Free online UUID generator: create RFC 4122 UUID v4 in bulk with a cryptographic RNG. Options for uppercase, no hyphens and braces, with one-click copy. - [Base Converter](https://qilingtools.com/en/base-convert): Free online number base converter: convert between binary, octal, decimal, hexadecimal and any base 2–36. Uses BigInt for huge integers, computed locally.