Technical SEO · AI crawler control
Paste your robots.txt, choose a URL path, and see how the file applies to major AI-related crawlers. The checker separates search/discovery bots from model-development crawlers, user-triggered fetchers, and policy tokens so you can make deliberate choices instead of blocking “AI” as one category.
Crawler names and provider policies last source-checked: October 5, 2026. Re-check provider documentation before production changes.
Review search/discovery crawlers separately from model-training crawlers. Blocking one does not necessarily block the other.
Use each provider’s documented model-development bot or product token where one exists.
robots.txt is not authentication. Protect confidential content with server-side access control, not crawler preferences.
AI crawler robots.txt checker
This runs locally in your browser. Your pasted robots.txt is not submitted anywhere by this page.
Scope: this is an RFC-style diagnostic for User-agent, Allow, and Disallow, including * and $ path matching. It does not fetch your live robots.txt, inspect HTTP/WAF access, authenticate bots, or guarantee that every provider implements every edge case identically.
AI crawler and user-agent reference
This list prioritizes crawler names and controls that are documented by the provider itself. “Search” and “training” are deliberately separated because several providers offer independent controls.
| Provider | Token | Purpose | robots.txt behavior / consequence | Official source |
|---|---|---|---|---|
| OpenAI | OAI-SearchBot | Search/discovery | Used for ChatGPT search. OpenAI says sites opted out will not be shown in ChatGPT search answers, though navigational links can still appear. | OpenAI |
| OpenAI | GPTBot | Model development | Used to crawl content that may be used to improve/train OpenAI generative foundation models. Control is independent from OAI-SearchBot. | OpenAI |
| OpenAI | ChatGPT-User | User-triggered | Used for certain user actions. OpenAI says robots.txt rules may not apply because requests can be user initiated; it is not the Search opt-out token. | OpenAI |
| Anthropic | Claude-SearchBot | Search/discovery | Used to improve Claude web-search result quality. Anthropic says blocking it can reduce visibility/accuracy in user search results. | Anthropic |
| Anthropic | ClaudeBot | Model development | Collects web content that may contribute to model training. Restricting it signals future material should be excluded from model-training datasets. | Anthropic |
| Anthropic | Claude-User | User-triggered | Supports user-directed retrieval. Anthropic documents it as a distinct control from automated model-development and search bots. | Anthropic |
| Perplexity | PerplexityBot | Search/discovery | Designed to surface/link sites in Perplexity search. Perplexity says it is not used to crawl content for foundation-model training. | Perplexity |
| Perplexity | Perplexity-User | User-triggered | Used for user actions; Perplexity says this fetcher generally ignores robots.txt. | Perplexity |
Googlebot | Search/discovery | Controls crawling for Google Search and related Search features. Blocking can affect Search eligibility. | ||
Google-Extended | Policy token | No separate HTTP crawler. Controls specified Gemini training/grounding uses; Google says it does not affect inclusion or ranking in Google Search. | ||
| Apple | Applebot | Search/discovery | Powers search technology across Apple experiences including Spotlight, Siri, and Safari. Applebot also may support generative AI context uses described by Apple. | Apple |
| Apple | Applebot-Extended | Policy token | Does not itself crawl pages. Lets publishers opt out of use of Applebot-crawled content for training Apple foundation models; blocking does not remove pages from Apple search. | Apple |
| Amazon | Amzn-SearchBot | Search/discovery | Used for search experiences in Amazon products such as Alexa. Amazon says it is not used for generative-model training. | Amazon |
| Amazon | Amazonbot | Model development | Used to improve Amazon products/services and may be used to train Amazon AI models. | Amazon |
| Amazon | Amzn-User | User-triggered | Supports user actions such as Alexa queries. Amazon says user-initiated requests may not follow all robots.txt directives. | Amazon |
| Common Crawl | CCBot | Open web dataset | Automated crawler for Common Crawl’s open web corpus. Common Crawl documents robots.txt blocking and dedicated verification information. | Common Crawl |
Example robots.txt policies
These are starting points, not universal production files. Merge them with your existing rules and sensitive-path restrictions. A provider-specific group does not automatically inherit your wildcard group.
Allow AI search, opt out of selected model-development uses
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Allow: /
User-agent: Amzn-SearchBot
Allow: /
User-agent: Amazonbot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot
Allow: /
User-agent: Applebot-Extended
Disallow: //private/, a newly created bot-specific group can become the applicable group for that bot. Add the appropriate sensitive-path restrictions to specific groups and test them. robots.txt still does not secure private content.Block a path for every ordinary crawler by default
User-agent: *
Disallow: /private/
Disallow: /account/
Disallow: /internal-search/Use authentication/authorization for genuinely private areas. A robots.txt path can reveal the existence of a URL and is not an access-control mechanism.
How this checker interprets robots.txt
The parser follows the core Robots Exclusion Protocol model: groups start with one or more User-agent lines followed by Allow/Disallow rules; multiple groups for the same specific token are merged; the wildcard group is used only when no specific group matches; and the most specific matching path wins. When equally specific Allow and Disallow rules conflict, the checker favors Allow.
The checker supports the protocol’s * wildcard and $ end-of-match character. It intentionally ignores non-access directives such as Sitemap for the allow/block calculation.
What it does not simulate: host/protocol scoping, robots.txt HTTP status handling, provider caching delays, CDN/WAF blocks, IP verification, JavaScript rendering, authentication, meta robots, X-Robots-Tag, or proprietary provider behavior beyond the documented caveats shown in the results.
Do not trust the user-agent string alone
User-agent strings can be spoofed. Where a provider publishes IP ranges or verification guidance, use those signals before labeling a server-log request as genuine.
- OpenAI: publishes separate IP-range endpoints for OAI-SearchBot, GPTBot, and ChatGPT-User.
- Perplexity: publishes IP JSON for PerplexityBot and Perplexity-User and recommends combining IP verification with user-agent matching at the WAF.
- Google: documents crawler verification and published IP ranges; Google-Extended itself is a robots token, not a distinct HTTP user agent.
- Apple: documents reverse-DNS verification and Applebot IP CIDRs.
- Amazon: publishes IP lists for Amazonbot, Amzn-SearchBot, and Amzn-User.
- Common Crawl: publishes CCBot address ranges and reverse-DNS guidance.
If robots.txt permits a crawler but the request still fails, continue with Search Counsel Co’s AI crawler access diagnostic to test HTTP response, WAF/CDN behavior, rendering, and logs.
Which AI crawlers should you allow or block?
| Your objective | Policy to review | What not to assume |
|---|---|---|
| Appear in ChatGPT search | Keep OAI-SearchBot crawlable and verify infrastructure access. | Allowing GPTBot is not required just because you want search visibility. |
| Appear in Claude web search | Review Claude-SearchBot separately from ClaudeBot. | Training and search are not the same control. |
| Appear in Perplexity search | Review PerplexityBot and WAF access. | Perplexity-User has different user-triggered behavior. |
| Stay eligible for Google Search/AI features | Do not block Googlebot for content you want indexed. | Blocking Google-Extended is not an opt-out from Google Search or its AI Search features. |
| Opt out of selected model-development crawling | Review documented training/model-development controls such as GPTBot, ClaudeBot, Amazonbot, Applebot-Extended, and Google-Extended. | No single “AI bot” rule covers every provider or use case. |
| Protect private/licensed material | Use authentication, authorization, server/CDN/WAF rules, and contractual controls where appropriate. | robots.txt is not security. |
For the conceptual difference between crawler rules and agent-oriented content guides, read llms.txt vs robots.txt. For a deeper policy setup, continue to robots.txt for the AI era. To measure citations and referral traffic after access is correct, use the AI visibility measurement guide.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT Search?
OpenAI documents GPTBot and OAI-SearchBot as independent controls. GPTBot is the model-development crawler; OAI-SearchBot is the Search crawler. Review both deliberately rather than treating GPTBot as the Search switch.
Does ChatGPT-User obey robots.txt?
OpenAI says robots.txt rules may not apply to ChatGPT-User because these fetches can be initiated by a user. Use OAI-SearchBot to manage automatic Search crawling and do not use robots.txt as a security boundary.
Does Perplexity-User obey robots.txt?
Perplexity states that Perplexity-User generally ignores robots.txt because it supports user-requested retrieval. PerplexityBot is the separate automated Search crawler.
Is Google-Extended the crawler for Google AI Overviews?
No. Google-Extended is a product token controlling specified Gemini training/grounding uses. Google says it does not affect inclusion or ranking in Google Search. Googlebot remains the crawler relevant to Search eligibility.
What is Applebot-Extended?
Apple describes Applebot-Extended as a secondary control for how Applebot-crawled data may be used to train Apple’s foundation models. Applebot-Extended does not itself crawl webpages, and blocking it does not remove pages from Apple search experiences.
Can I block model training but allow AI search?
For providers that document separate controls, yes. For example, OpenAI separates OAI-SearchBot from GPTBot, Anthropic separates Claude-SearchBot from ClaudeBot, and Amazon separates Amzn-SearchBot from Amazonbot.
Does an “Allowed” result mean the bot can definitely fetch my page?
No. robots.txt is only one layer. A WAF, CDN rule, rate limit, login, server error, or rendering problem can still prevent access. Verify the request path and infrastructure separately.
Can robots.txt protect private information?
No. The Robots Exclusion Protocol is not access authorization. Use authentication and server-side controls for content that must remain private.
Primary sources and maintenance policy
The reference above was checked against current provider documentation on October 5, 2026: OpenAI crawlers, Anthropic crawlers, Perplexity crawlers, Google crawlers and product tokens, Applebot, Amazonbot, Common Crawl CCBot, and RFC 9309.
Provider names, IP ranges, product behavior, and control semantics can change. This page should be reviewed when a provider publishes a crawler-policy update and at least quarterly even when no change is announced.
The next layer is not another robots rule. Check HTTP status, WAF/CDN policies, rendering, and verified request logs with the AI crawler access diagnostic. For broader strategy, see Search Counsel Co’s GEO evidence guide.