By Rahul Saini, Author at Search Counsel Co. Last updated August 2026.

1) robots.txt vs llms.txt: what’s the difference?

They do different jobs, and mixing them up is the most common mistake in AI crawler setup.

robots.txt is about access. It sits at your site root and tells crawlers which user-agents may fetch which paths. This is the file you use to allow or block GPTBot, ClaudeBot, PerplexityBot, and the rest. It has been the web standard for decades and every major AI company honors it for its declared crawlers. Our companion guide, robots.txt for the AI era, covers the enforcement side in more technical depth.

llms.txt is about curation. It’s a Markdown file, also at your root, that hands AI systems a short, ranked list of your most useful pages so they don’t have to guess. It doesn’t grant or deny access. It says “if you’re going to read me, start here.”

You’ll see guides that tell you to put lines like User-agent: GPTBot / Disallow: / inside llms.txt. That’s wrong. Access rules belong in robots.txt. llms.txt is a content map, not a rules file, and loading it with crawler directives just breaks the format.

2) The AI crawlers that matter in 2026

You can’t write good rules until you know what you’re writing them for. Here are the crawlers worth knowing, grouped by who runs them. Each user-agent is its own independent switch, so allowing one from a company does not automatically allow the others.

Two things change over time here: the exact token strings and which company owns which bot. Anthropic, for example, consolidated its older anthropic-ai and claude-web tokens into the three above. Always confirm against the vendor’s current crawler docs before you ship a rule. If crawler volume is straining your server, our guide to crawl budget covers the load side.

3) Training, retrieval, or user-triggered: three jobs, three decisions

The single most useful idea in this whole topic is that “AI crawler” isn’t one thing. The bots fall into three groups, and each group is a separate decision.

• Training crawlers collect content to build or improve models: GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, Bytespider, Meta-ExternalAgent. Block these and your content won’t feed future models. There’s no direct traffic either way, so this is a values and licensing decision, not a visibility one.
• Retrieval crawlers fetch pages to cite in live answers: OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Google’s own Googlebot. These are the ones that get you named and linked inside AI answers. Block one and you disappear from that engine’s responses. Almost every business wants these allowed.
• User-triggered fetchers visit a page because a human asked, in real time: ChatGPT-User, Claude-User, Perplexity-User. These often ignore robots.txt on the logic that a person requested that specific page. Perplexity states plainly that its user-triggered fetches generally do not follow robots.txt.

4) Setting up robots.txt for AI crawlers

Your robots.txt lives at yourdomain.com/robots.txt, at the root, nowhere else. One thing to keep in mind before you start: under the formal standard (RFC 9309), robots.txt is an instruction, not a lock. Compliant crawlers obey it. It does not physically stop anyone, so it isn’t a security tool. For that you need real access controls, which we’ll cover below. It’s also worth remembering that blocking a page here doesn’t remove it from an index; that’s what noindex is for.

Here are three setups for the three situations most sites are in. Bot-specific groups override the wildcard, so list them after your User-agent: * block.

Setup A. Maximum AI visibility (the right default for most businesses). Let AI crawlers reach everything public, and only fence off private areas.

# Let all crawlers reach public content
User-agent: *
Allow: /

# Keep private areas out of everything
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/

Sitemap: https://yourdomain.com/sitemap.xml

Setup B. Stay citable, opt out of training. This is the nuanced middle ground: appear in AI answers, but keep your content out of model training.

# Allow AI search and retrieval crawlers (stay citable)
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Allow user-triggered fetchers
User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Perplexity-User
Allow: /

# Opt out of AI model training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

# Everything else, including Googlebot for Search and AI Overviews
User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Setup C. Block AI as far as robots.txt allows. For paywalled, licensed, or proprietary content. Note that Googlebot stays allowed so you remain in Google Search, which also keeps you in AI Overviews. There’s no clean way around that, as the next section explains.

# Block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

# Block AI search and retrieval crawlers
User-agent: OAI-SearchBot
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

# Keep Googlebot so you stay in Google Search
User-agent: Googlebot
Allow: /

User-agent: *
Allow: /

After any change, give it time. Bots re-fetch robots.txt on their own schedule, and a new rule typically takes roughly 24 to 72 hours to register across the major engines.

5) The Google-Extended and AI Overviews trap

Here’s the point that trips up even enterprise teams: Google-Extended does not stop AI Overviews.

Google-Extended isn’t a crawler with its own identity. It’s a permission token that controls whether Google uses your content to train Gemini and for grounding in Vertex AI. Blocking it keeps your content out of Google’s AI training. Google states in writing that this has no effect on your rankings or your inclusion in Search.

But AI Overviews and AI Mode are Search features. They run on Google’s regular index, fetched by Googlebot, not by Google-Extended. So the only ways to leave AI Overviews are to block Googlebot, which removes you from Google Search entirely, or to use the nosnippet meta tag, which also strips your normal search snippets and tanks your click-through rate. Neither is worth it for a business that wants to be found.

On the training side, a December 2025 BuzzStream study of the top 100 US and UK news sites found Google-Extended was the least-blocked of the training bots, a sign that even cautious publishers are wary of doing anything that looks like antagonizing Google. For where each engine’s citations actually come from, see our guide on how ChatGPT, Perplexity, and Gemini each choose their sources.

6) How to write an llms.txt file

llms.txt is a Markdown file at yourdomain.com/llms.txt that lists your most important pages so AI systems can find your best content without crawling everything. It was proposed by Jeremy Howard of Answer.AI in September 2024 and has spread quickly as a convention.

The structure is short: a top-level heading with your site name, a one-paragraph summary in a blockquote, then sections of links with a brief description on each. A minimal version looks like this:

# Search Counsel Co.

> AI search and SEO agency. We help brands get cited by
> ChatGPT, Perplexity, Gemini, and Google AI, and rank in
> traditional search.

## Core guides
- [What is GEO](https://searchcounselco.com/generative-engine-optimization/): Plain-English guide to generative engine optimization.
- [AI crawler setup](https://searchcounselco.com/lms-txt-robots-txt/): robots.txt and llms.txt for AI crawlers.

## Tools
- [AI Visibility Checker](https://searchcounselco.com/): See whether AI engines cite your brand.

A few rules that keep it valid: place it at the root, serve it over HTTPS as plain text, and keep it a curated shortlist rather than a dump of every URL. That full list is what sitemap.xml is for. If you want a longer, full-content version, the convention is a separate llms-full.txt.

7) Should you allow or block AI crawlers?

For most businesses that want to be found, allow the search and retrieval crawlers. That is how you get cited in ChatGPT, Perplexity, and Google’s AI answers, which is the fastest-growing discovery channel there is. Then decide on the training crawlers separately, based on how you feel about your content feeding models.

Blocking everything with a blanket Disallow: / is the opposite of what you want if AI visibility matters. It’s a defense a public marketing site doesn’t need, and it quietly costs you citations.

Blocking training crawlers makes sense for a narrower group: publishers with licensed or premium content, sites behind paywalls, regulated industries where content misuse creates liability, and anyone planning to license their content to AI companies for compensation. If that’s you, Setup B or C above is the starting point.

The data shows this split in practice. In that December 2025 BuzzStream study, 79% of top news sites blocked at least one AI training bot and 71% blocked at least one retrieval bot, yet only 14% blocked every AI bot and 18% blocked none. News publishers protect exclusive content aggressively. Most businesses chasing reach should not copy them wholesale. Separately, Cloudflare began blocking AI crawlers by default on new domains from July 1, 2025 and launched a Pay Per Crawl option, which is a strong reason to actually check what your own CDN is doing before you assume your robots.txt is the final word.

8) Common mistakes (and the WAF gotcha)

Most AI crawler problems come from a short list of avoidable errors.

• Blocking GPTBot and thinking you’ve left ChatGPT search. You haven’t. GPTBot is training only. OAI-SearchBot is the crawler behind ChatGPT search citations, and it’s a separate switch.
• Allowing a bot in robots.txt but blocking it at the CDN or WAF. This is the big one. A firewall rule or rate limit that returns 429 Too Many Requests to the crawler overrides your friendly robots.txt. In 2024 and 2025 this was the most common reason sites missed ChatGPT citations despite an open crawler policy.
• A broad Disallow or wildcard that swallows the bots you meant to allow. One stray rule at the top of the file can catch everything below it.
• Putting access rules inside llms.txt. User-agent and Disallow lines belong in robots.txt. llms.txt is a content map.
• Expecting robots.txt to enforce anything. Compliant crawlers from OpenAI, Anthropic, Google, and Perplexity honor it. Bad actors and user-triggered fetchers may not. Real protection needs authentication and a firewall, not a text file.

The rendering side of access, JavaScript content and server response, sits alongside this. If you want the deeper version, see our guide on how to make sure AI crawlers can access and read your site.

9) How to verify your setup is working

Shipping the files is step one. Confirming they do what you intended is step two.

• Load both files in a browser. Visit yourdomain.com/robots.txt and yourdomain.com/llms.txt. Each should return a 200 status and plain readable text, with no redirect chain.
• Read your server logs. Filter for the AI user-agents (GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Googlebot) and check who is actually visiting, how often, and what status codes they receive.
• Verify identity by IP where you can. OpenAI, Google, Perplexity, and Common Crawl publish IP range files you can match requests against. Anthropic recommends robots.txt control and reverse DNS instead. This matters because spoofers fake user-agent strings to look like a trusted bot.
• Track referrals. ChatGPT tags outbound links with the parameter utm_source=chatgpt.com, so you can see ChatGPT-driven visits in GA4 and confirm citations are turning into traffic.
• Test the actual outcome. Ask ChatGPT, Perplexity, and Gemini your target questions and see whether you get named. Our AI Visibility Checker automates this across a set of prompts, and tracking AI visibility covers how to run it as a routine.

Give changes 24 to 72 hours to register before you judge whether a rule worked.

Sources used for this guide

Crawler behavior changes fast, so this guide leans on the vendors’ own documentation and named studies rather than secondhand claims.

FAQ: robots.txt and llms.txt for AI

Conclusion: two files, two jobs, one setup

AI crawler setup comes down to getting two files right in the right order. Start with robots.txt: allow the retrieval crawlers so you stay citable, and make training a separate, deliberate choice. Add a tidy llms.txt as a low-cost hedge, knowing it isn’t yet doing much on its own. Then check that nothing at your CDN or firewall is quietly blocking the bots you allowed, because that hidden 429 is the reason most sites go missing from AI answers.

With access sorted, the next lever is structure. Machines cite what they can cleanly extract, so move on to the best schema for AI citations and how to make sure AI crawlers can access and read your site, both part of our AI search optimization guide.