By Rahul Saini, Author at Search Counsel Co. Last updated [JULY] 2026.

1) What robots.txt is and how it works

robots.txt is a plain text file that lives at the root of your site, at yourdomain.com/robots.txt, and it’s the first thing most crawlers check when they arrive. It uses a simple format: a User-agent line names a bot, Disallow lines list paths that bot shouldn’t crawl, Allow lines carve out exceptions, and a Sitemap line points to your sitemap. Formalized as the standard RFC 9309, it’s how you tell crawlers which parts of your site they may fetch, and it’s one of the first controls in how search engines work.

Two points are worth fixing in your mind. First, robots.txt controls crawling, not indexing. A page you block in robots.txt can still appear in search results as a bare URL if other pages link to it, because blocking crawling isn’t the same as removing something from an index. To keep a page out of the index, you need a noindex tag, which is covered in our guide to fixing indexing problems. Second, don’t block the CSS and JavaScript files that render your pages, or crawlers can’t see your content, a point our guide to Google’s December 2025 rendering update gets into. For AI crawlers specifically, the golden rule is to name each bot on its own line rather than relying on a single catch-all, because that’s the only way to make the precise choices the next sections describe.

2) The AI crawlers you need to know

Until 2023, “the crawler” basically meant Googlebot. That changed when generative AI arrived. OpenAI shipped GPTBot in August 2023, Google split AI training out of Googlebot with Google-Extended a month later, and today every major AI provider runs at least one documented crawler. AI crawler traffic has since grown enormously, to the point where it rivals traditional search crawling on many sites, a trend covered in our guide to crawl budget. The important part isn’t the volume, though. It’s that these bots do different jobs, and they fall into three groups:

• Training crawlers collect your pages in bulk to help build a model. They run constantly, aren’t triggered by any user, and send no traffic back to you.
• Retrieval and search crawlers fetch pages to answer questions in AI search, and they cite the source with a link, which can send you visitors.
• User-triggered fetchers grab one page because a person asked an assistant to read or summarize it.

The crucial thing is that the big vendors run separate bots for these jobs, and blocking one does not block the others. Here are the ones that matter most, and our guide to AI crawler access goes further on managing them:

The one to burn into memory: GPTBot is not OAI-SearchBot, and ClaudeBot is not Claude-SearchBot. Blocking the training bot does not block the search bot, and blocking the search bot removes you from that engine’s answers entirely.

3) The choice that matters: training vs visibility

Because the bots are split by purpose, your decision splits too. There are really three positions, and picking one is mostly about weighing AI visibility against control over your content:

For most businesses that want to be found, the middle position is the sensible default, and the economics make it easy. Training crawlers do the vast majority of the crawling but send almost no visitors: analysis from Cloudflare found some training bots making tens of thousands of page fetches for every single referral visit. So blocking training costs you next to nothing in traffic, while keeping the retrieval bots open preserves one of the fastest-growing referral channels there is, one you can measure by tracking ChatGPT and Perplexity referrals in GA4. The mistake to avoid is the blanket block. Studies of top news publishers found that while most block training bots, roughly seven in ten also block a retrieval or search bot, usually while intending only to opt out of training, which quietly removes them from ChatGPT or Perplexity answers. Research from Rutgers and Wharton in late 2025 found that publishers blocking AI crawlers saw a meaningful drop in total traffic without reliably reducing how often they were cited.

One reassurance worth repeating, because it stops people short: opting out of AI training does not touch your Google rankings. Google has confirmed that blocking Google-Extended has zero effect on Googlebot or search position; it only controls whether your content trains Gemini. The same separation holds across the other vendors. Blocking training is a content-policy decision, not an SEO one. If you have strong content-licensing prospects or work in a sensitive category, blocking training crawlers is a legitimate stance, just make it on purpose.

4) How to write robots.txt for AI crawlers

The syntax is the same as always: name a user-agent, then Allow or Disallow. Here are the three configurations that match the three goals. First, the welcome mat, allowing everything:

Second, the common middle ground, allowing the bots that cite you while opting out of training:

Third, to keep AI out entirely, give each of those crawlers a Disallow, and remember to back it up at the CDN, since robots.txt alone won’t stop a determined bot. A few practical notes: always include the Sitemap line, list each bot explicitly rather than trusting a wildcard, and revisit the file periodically, because vendors add and rename crawlers, and older strings can quietly stop matching the current bot.

5) robots.txt is a request, not a lock

This is the part that keeps people out of trouble. robots.txt is a voluntary standard, not a technical barrier and not a law. It works because well-behaved crawlers choose to honor it, which the major AI companies generally do, and you can confirm it in your own server logs. But three honest caveats apply.

Some crawlers simply ignore it. ByteDance’s Bytespider has a long record of doing so, and in August 2025 Cloudflare published evidence that Perplexity was using undeclared crawlers that rotate user-agents and addresses to slip past robots.txt directives. On top of that, a user-agent can be faked, so the only reliable way to confirm a bot is who it claims to be is a reverse DNS check, not the header. And user-triggered fetches sit in a gray area: when a person asks an assistant to read a specific URL, vendors often treat that as the user accessing the page rather than the bot crawling it, so your crawl rules may not apply the way you expect. The practical conclusion is that if you truly need to stop a bot, you enforce it at the server or CDN level, with something like Cloudflare’s AI crawler controls or a firewall rule, not in robots.txt.

You may also come across llms.txt, a newer file meant to guide AI systems to your most useful pages. It’s worth understanding for what it is: a navigation aid, not an access control. robots.txt governs access; llms.txt suggests where to look. Adoption is still early and no major vendor has confirmed it changes their behavior, so treat it as a low-risk experiment rather than a lever, and never as a way to opt out of training. We cover it as part of the bigger picture in our guide to machine-first architecture.

6) Common mistakes and how to test

Most robots.txt problems in the AI era come down to a handful of errors:

• Blocking retrieval bots by accident. The big one. A blanket block of AI user-agents, or an over-eager CDN rule, removes you from AI answers you wanted to be in.
• Using robots.txt to deindex. Blocking a page doesn’t remove it from search; it can still show as a bare URL. Use noindex instead.
• Blocking rendering files. Disallowing the CSS and JavaScript that build your pages stops crawlers seeing your content, as our guide to JavaScript SEO explains.
• Blocking only old crawler names. Vendors rename bots, and outdated strings stop matching. If your rules reference a retired user-agent, they may not be doing anything.
• Assuming the file is enforcement. It’s a request. For bots that ignore it, only your server or CDN can actually stop them.

To check your work, view your live file at yourdomain.com/robots.txt, use the robots.txt tester in Search Console to confirm your rules parse, and then read your server logs to see which AI bots are actually visiting and whether they’re respecting your rules. Logs are the ground truth here, since they show real behavior rather than intent. If bot traffic is straining your server, our guide to site speed optimization covers how to serve them without slowing real visitors down, and our guide to fixing crawl errors covers what to do when they hit problems.

7) Sources used for this guide

This guide draws on the AI vendors’ crawler documentation and independent analysis of crawler behavior. Some figures are noted as directional.

FAQ: robots.txt and AI crawlers

Conclusion: make two decisions, not one

The old instinct with AI bots, block them all or allow them all, no longer fits, because the bots no longer do one job. The move in 2026 is to make two separate decisions: whether to be visible in AI answers, where the default for most sites is yes, and whether to let your content train models, which is a genuine content-policy call. Set them independently in robots.txt, remember it opts you out of AI training without touching Google, and back up anything that truly must be blocked at your CDN, because the file is a request, not a wall.

Controlling crawlers is one half of getting ready for AI. The other half is making your content legible to the ones you welcome, through schema markup and the broader work of machine-first architecture, all under the technical SEO pillar. For the content side of earning those citations, start with our AI search optimization guide.