Technical SEO · AI crawler configuration
llms.txt vs robots.txt: What Each File Does in 2026
robots.txt tells cooperating crawlers which URLs they may fetch. llms.txt gives agents that choose to use it a curated Markdown guide to useful pages. Neither file acts as a password or firewall. If your goal is AI search visibility, first make sure the appropriate search crawler can access readable, indexable public content. Add llms.txt only when a maintained content guide will help a real agent workflow.
/robots.txt for crawler preferences, /llms.txt for an optional content map, /sitemap.xml for URL discovery, and server/CDN/authentication controls for actual access enforcement. A correctly formatted llms.txt does not grant permission to crawl a disallowed page, cause Google to rank it, or guarantee an AI citation.| Layer | Question it answers | Does it enforce access? | Priority |
|---|---|---|---|
| robots.txt | May a compliant bot crawl this path? | No. It publishes a crawl preference. | Review before changing crawler policy. |
| llms.txt | Which resources should a compatible agent read? | No. It is a proposed Markdown convention. | Optional for curated documentation and agent workflows. |
| sitemap.xml | Which URLs should search engines discover? | No. It is a discovery aid. | Keep accurate and current. |
| Server, authentication, CDN/WAF | Can this request actually retrieve content? | Yes, when properly configured. | Use for real protection and access diagnosis. |
1. llms.txt vs robots.txt: the actual difference
robots.txt is a file at the root of a particular host and protocol, such as https://example.com/robots.txt. Its User-agent, Allow, and Disallow directives express crawling preferences for cooperating bots. A crawler may still receive a 403 or 429 from the infrastructure even when robots.txt allows its request. Blocking crawling is also not the same as reliably removing a URL from Google Search. See Google's robots.txt introduction.
llms.txt is an optional Markdown file, usually at https://example.com/llms.txt. It describes a project and links to selected, useful resources. Its readers may include software agents or documentation tools that deliberately choose to fetch it. It cannot set crawl permissions, override robots.txt, secure private content, or compel any AI provider to cite the site. The format is documented in the llms.txt proposal.
What about sitemap.xml?
An XML sitemap helps search engines discover the site's intended URLs. It is not a bot permission list or a curated explanation of which documents an AI agent should read first. Maintaining a good sitemap and a concise llms.txt can be compatible, but the sitemap should not be copied wholesale into llms.txt.
2. llms.txt v2: what changed in August 2026?
The August 2026 revision of the proposal clarifies that /docs/llms.txt can describe a documentation subpath and that the most specific applicable file should be used. It also proposes discoverability links from an HTML page or HTTP header to its Markdown alternative or covering llms.txt file.
| Feature | Practical meaning |
|---|---|
| Root or subpath file | Publish /llms.txt sitewide or a focused /docs/llms.txt for that section. |
| Markdown alternatives | A page may expose a clean Markdown representation where useful and maintained. |
| Discovery links | rel="alternate" can identify a Markdown version; rel="describedby" can identify the covering llms.txt file. |
| Optional links | An ## Optional section remains a convention for secondary resources, not a universal parser command. |
These are additions to a proposal, not proof of support by every major search or answer engine. Only advertise a Markdown URL that really exists and serves the correct content; do not add placeholder alternate links to a website template.
3. Documented behavior vs assumptions
| Statement | Evidence status |
|---|---|
| OpenAI documents separate search and model-development crawlers. | Documented by OpenAI. |
| Google Search does not use llms.txt as a visibility or ranking requirement. | Explicitly documented by Google. |
| llms.txt v2 describes subpath files and discovery links. | Specified in an external proposal; provider adoption varies. |
| Publishing llms.txt increases AI citations or organic rankings. | Not established by the cited primary sources. |
| Allowing a search crawler guarantees it will fetch or cite a page. | False: permission and actual retrieval/surfacing are distinct. |
4. What major providers document about their crawlers
Names and purposes below are from the providers' published documentation, not an observation of which bots have visited this website.
| Provider | Search or discovery | Other documented roles |
|---|---|---|
| OpenAI | OAI-SearchBot | GPTBot for model development; ChatGPT-User for certain user actions. |
| Anthropic | Claude-SearchBot | ClaudeBot for model development; Claude-User for user-initiated retrieval. |
| Perplexity | PerplexityBot | Perplexity-User for user-initiated requests. |
| Google Search | Googlebot | Google-Extended is a separate robots.txt product token, not a distinct crawling user-agent. |
Check current instructions before deploying rules: OpenAI crawler documentation, Anthropic crawler documentation, Perplexity crawler documentation, and Google's crawler and product-token reference.
5. Training, search, and user-triggered fetching are different decisions
A publisher can allow a provider's search bot while setting a different rule for its model-development bot. For instance, OpenAI documents separate OAI-SearchBot and GPTBot settings. Blocking one does not automatically block the other. A user-triggered fetcher is different again: OpenAI notes that robots.txt rules may not apply to some ChatGPT-User requests; Perplexity says its user-requested fetcher generally ignores them. Do not rely on a robots rule to protect confidential documents.
6. Which AI crawlers should you allow or block?
| Your objective | What to review |
|---|---|
| Make public pages eligible for ChatGPT search discovery | OAI-SearchBot robots policy and CDN/IP access. |
| Let Claude or Perplexity discover public pages | Claude-SearchBot and PerplexityBot access separately. |
| Express a model-training preference | The relevant provider's training bot or product token, separately from search access. |
| Stay eligible for Google Search and its AI features | Googlebot access, indexability, readable text and normal Search preview controls. |
| Restrict confidential or licensed material | Authentication, authorization, server/CDN rules and contracts as appropriate; robots.txt alone is not protection. |
Do not copy a long allow/block template blindly. Bot-specific groups can affect how a crawler interprets wildcard rules; test the effective policy for every important path. For detailed configurations, use Search Counsel Co's robots.txt for AI crawlers guide.
7. Does llms.txt actually improve AI search or SEO?
There is no demonstrated, general ranking or AI citation lift from uploading an llms.txt file alone. It can still have a clear operational use: an agent explicitly designed to read a documentation index may use it to find the right guide or Markdown page. This is a narrower claim than saying every AI search engine reads the file, and it depends on the agent's behavior.
For a small marketing website, prioritize accurate public pages, crawlability, internal linking and measurement. For complex technical documentation, a well-maintained llms.txt can be a reasonable, separately measurable experiment. Check actual server logs for verified requests to the file before claiming adoption on your own site.
8. Does Google Search use llms.txt for AI Overviews?
No. Google's official AI optimization guidance states that Google Search does not use llms.txt files as a special mechanism for Search or its generative AI features. Google says you do not need new AI text files or special markup to appear in AI Overviews or AI Mode. An ordinary file may still be discovered or indexed as a file; that does not make it a special ranking signal.
For Google AI feature eligibility, focus on Googlebot crawl access, indexability, content available in text, helpful internal links and appropriate preview controls. Google does not guarantee indexing, ranking or inclusion in an AI feature. The narrower Google-only question is covered in Do You Need llms.txt for Google AI Search?.
9. Googlebot, Google-Extended and snippet controls
Googlebot crawls pages for Google Search. Google-Extended is a separate robots.txt product token that manages specified Google AI training and grounding uses; Google says the token does not affect inclusion or ranking in Google Search. Blocking Google-Extended is not an opt-out from AI Overviews or AI Mode, which are Search features. See Google's official definition.
Google describes nosnippet, max-snippet, data-nosnippet and noindex as relevant Search-content and preview controls; each has broader trade-offs, so do not deploy them merely to change an AI appearance without reviewing the effects on ordinary Search.
10. How to create robots.txt and llms.txt safely
A. Decide the crawler policy before editing robots.txt
This illustrative fragment expresses an opt-out for GPTBot while leaving OAI-SearchBot allowed. It is not a complete production robots.txt for every website. Merge it with existing groups after checking your sensitive paths, host variants and provider policy.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /Because a specific user-agent group can take precedence over a wildcard group's rules, do not assume a wildcard Disallow: /private/ will remain effective for a separately named bot. Add any path restrictions appropriate to that named group and test them. Private content needs authentication regardless. Before saving, use the robots.txt specification and parser guidance and confirm the current live file can be safely merged.
B. Make a short llms.txt content guide
Publish plain Markdown at https://yourdomain.example/llms.txt, with a project heading, description and curated links. This example uses real Search Counsel Co resources, but is an example, not a claim that it has been published to the site's root:
# Search Counsel Co
> Technical SEO, AI search and organic visibility research.
## Core guides
- [llms.txt vs robots.txt](https://searchcounselco.com/lms-txt-robots-txt/): Understand the two files and practical configuration choices.
- [AI-era robots.txt](https://searchcounselco.com/robots-txt-ai-era/): Review crawler-specific policies.
- [AI Crawler Access](https://searchcounselco.com/ai-crawler-access/): Diagnose HTTP, WAF and rendering issues.
## Optional
- [Tracking AI Visibility](https://searchcounselco.com/track-ai-visibility/): Measure citations and referrals separately.Keep the file public, concise, accurate and synchronized with the URLs it references. Do not include unpublished, customer-specific or authenticated resources. An agent that does not seek out llms.txt will not benefit just because it exists.
C. Optional v2 discovery: add only real Markdown alternatives
If you maintain a valid Markdown version for an actual page, its page template or HTTP headers may advertise that URL. The v2 proposal also describes identifying the covering llms.txt file:
<!-- Example only: put in the HTML head or matching HTTP Link header, not in this article body. -->
<link rel="alternate" type="text/markdown" href="/docs/example.md">
<link rel="describedby" type="text/markdown" href="/docs/llms.txt">Do not paste those sample links into your current page: the example resources may not exist on your site, and this proposal is not a Google Search requirement.
11. Who needs llms.txt? A practical decision rule
Consider it if your documentation, API reference or knowledge base has many high-value pages, and you can maintain an agent-oriented map with real Markdown links. Defer it if the website has unresolved indexing problems, weak core content or no plausible workflow that reads the file. A local business or straightforward brochure site generally has more important technical and content work to complete first.
Do not assume that adding a file will improve Google AI visibility. Choose a specific use case, such as helping an internal coding agent locate the correct API documentation, then measure whether that agent retrieves and successfully uses the resources.
12. How to verify the setup after deployment
- Inspect the existing robots file. Open
https://yourdomain.example/robots.txtand test the effective rule for each important user agent and path. A missing file normally means no robots restrictions, not automatic blocking. - Check ordinary HTTP access. Confirm intended public URLs and llms.txt return the correct content and status. Investigate redirects, 403 challenges, 429 rate limits and server errors; a 200 can still contain an empty JavaScript shell.
- Inspect the actual HTML. Verify that the page's main explanation and relevant links are readable without requiring a complex user interaction.
- Check the WAF and verified bot logs. User-agent strings are spoofable. Match visits against the provider's documented verification process or published IP ranges before reporting genuine crawler visits.
- Check Google Search Console separately. Use URL Inspection for index and selected-canonical signals; track ordinary Web search metrics independently of your server-log measures and manual AI citation observations.
13. Common mistakes to avoid
- Putting
User-agentorDisallowin llms.txt rather than robots.txt. - Calling robots.txt a security boundary or using it to hide private URLs.
- Assuming a bot-specific
Allow: /automatically inherits every wildcard restriction. - Blocking GPTBot and assuming OAI-SearchBot is blocked too, or confusing Google-Extended with Googlebot.
- Claiming a website's llms.txt improves citations without verified use or a defensible comparison.
- Publishing broken Markdown alternatives, private links or a full sitemap dump as llms.txt.
- Reading an isolated first-page impression as proof of a stable first-page ranking.
14. Frequently asked questions
Is llms.txt the same as robots.txt?
No. robots.txt publishes crawling preferences for compliant bots. llms.txt optionally curates resources for agents that choose to read it.
Does llms.txt replace robots.txt or sitemap.xml?
No. All three have distinct purposes: crawl rules, a curated content guide and URL discovery, respectively.
Does llms.txt help Google rankings or AI Overviews?
Google says Search does not use the file as a special visibility mechanism. Publishing it does not make a page eligible for inclusion automatically or prove a ranking benefit.
Can I allow search while expressing a training opt-out?
Some providers document separate crawler or product-token controls for these uses. Review the exact vendor policy and test every important URL; do not assume the same split applies to every provider.
Does robots.txt prevent an AI assistant from reading private information?
No. User-triggered requests and noncompliant clients can behave differently. Use authentication, authorization and genuine server-side access controls for private information.
Can llms.txt live in a subdirectory?
Yes, the August 2026 v2 proposal defines subpath-specific llms.txt files, such as /docs/llms.txt. That proposal does not require every agent to fetch them.
What is llms-full.txt?
It is a related convention some sites use to provide expanded text in one file; it is not a replacement for robots.txt and should not be assumed to be read by every AI service.
How can I confirm an AI search bot really fetched my content?
Review raw request logs, HTTP status and the provider's current bot-verification information. A matching user-agent string alone is not proof of identity or a future citation.
15. Primary sources and how this guide was checked
Technical claims above are grounded in the llms.txt proposal and its August 2026 v2 changes; the Robots Exclusion Protocol; OpenAI, Anthropic, and Perplexity crawler documentation; and Google's AI Search guidance and crawler/product-token documentation. A proposal's intended behavior is not proof that a given vendor follows it; actual visits must be confirmed in site logs.