AI Search Optimization Guide
How to Make Sure AI Crawlers Can Access and Read Your Site (2026)
Getting cited by AI starts with two things a machine needs: reaching your page, and reading the content once it’s there. robots.txt is only the first gate. The bigger, hidden problems are firewalls that block bots and JavaScript that hides your content. Here’s how to make sure GPTBot, ClaudeBot, and PerplexityBot can actually see your pages.
By Rahul Saini, Author at Search Counsel Co. Last updated August 2026.
The short answer
AI crawlers can fail to see your content for two reasons. Access: a firewall, CDN, or rate limit blocks them even when robots.txt allows them, or the page returns an error. Reading: almost no AI crawler runs JavaScript, so any content that only appears after scripts execute is invisible to them, even if the page ranks fine on Google, which does render. The fix is to serve your important content in the initial HTML through server-side rendering, static generation, or prerendering, make sure your infrastructure isn’t quietly blocking AI bots, and confirm every key page returns a clean, full-HTML response at a real URL.
Which crawlers render JavaScript
| Crawler | Company | Renders JavaScript? |
|---|---|---|
| Googlebot | Yes (also powers AI Overviews and AI Mode) | |
| Gemini | Yes (uses Googlebot’s rendering engine) | |
| Applebot | Apple | Yes |
| GPTBot, OAI-SearchBot, ChatGPT-User | OpenAI | No |
| ClaudeBot, Claude-SearchBot, Claude-User | Anthropic | No |
| PerplexityBot | Perplexity | No |
| Bingbot | Microsoft | Limited |
| Bytespider, Meta-ExternalAgent | ByteDance, Meta | No |
Jump to what you need
Article note: Written by Rahul Saini at Search Counsel Co. The rendering findings here are drawn from Vercel and MERJ’s crawler analysis and corroborating 2026 studies, plus the vendors’ own crawler docs. Crawler behavior changes, so re-verify against current documentation and your own server logs before you rely on any single point.
1) The two things machines need: reach and read
Before an AI engine can cite you, two simple things have to be true. It has to reach your page, meaning nothing blocks the request. And it has to read the content, meaning the words are actually in what it fetches. Miss either one and you’re invisible, no matter how good the content is.
These are separate problems with separate fixes, and a page can pass one while failing the other. robots.txt, which we cover in the complete AI crawler setup, is only the first gate on the reach side. Plenty of sites clear robots.txt and still fail on infrastructure or on rendering.
Simple rule: AI can only cite what it can reach and read. Fix access first, then readability. A page that ranks on Google can still fail both.
2) Access: what blocks AI crawlers (beyond robots.txt)
robots.txt tells crawlers what they may fetch, but plenty of sites block AI bots without ever touching it. The usual culprits sit lower in your stack.
- Firewall, CDN, or bot management. This is the big one. A WAF rule or rate limit that returns 429 Too Many Requests overrides your friendly robots.txt. In 2024 and 2025 this was the most common reason sites missed ChatGPT citations despite an open crawler policy. Cloudflare raised the stakes by starting to block AI crawlers by default on new domains from July 1, 2025, so many sites now block AI bots and don’t know it.
- Error responses. AI crawlers hit a lot of 404s, and they crawl far less often than Googlebot, so a broken URL or a redirect chain can keep a page out of AI answers for weeks. Keep redirects clean, sitemaps current, and URLs consistent, and work through any crawl errors you find.
- Over-aggressive bot blocking. Verifying crawlers by IP is smart, since AI bots run from US data centers and you can confirm them by published IP ranges or reverse DNS. The risk is a rule tuned so tightly it turns away the real crawlers too.
The check is quick: read your server logs, filter for the AI user-agents, and look at the status codes they get back. If GPTBot or OAI-SearchBot is receiving 403s or 429s, your infrastructure is the problem, not your content. Our companion guide, robots.txt for the AI era, covers the enforcement layer in more depth.
From experience: when a client isn’t showing up in ChatGPT, the cause is rarely the writing. It’s a firewall or rate limit quietly returning 429 to the crawler, or a page that 404s. Read the logs before you rewrite anything.
3) Reading: the JavaScript rendering gap
This is the one that catches good sites off guard. Almost no AI crawler runs JavaScript. When GPTBot, ClaudeBot, or PerplexityBot fetches a page, it reads the raw HTML the server sends and stops there. It doesn’t execute scripts, wait for data to load, build the page, or click anything. So any content that only appears after JavaScript runs is invisible to it.
The data is hard to argue with. Vercel and MERJ analyzed more than 500 million GPTBot fetches and found zero evidence of JavaScript execution. The crawlers do download JavaScript files, ChatGPT on about 11.5% of requests and Claude on about 23.8%, but they read the code as text, they don’t run it. The same pattern held for Anthropic’s, Meta’s, ByteDance’s, and Perplexity’s crawlers.
Here’s the trap. Google renders JavaScript. It fetches your page, runs the scripts, builds the full page, and indexes the result, so a client-side-rendered page can rank number one on Google. That pipeline changed shape again with Google’s December 2025 rendering update, but the principle holds: Google renders, the AI crawlers read only the raw HTML, so the exact same page can be a blank shell to them. Your rankings look healthy, your Core Web Vitals pass, your SEO audit comes back clean, and ChatGPT still can’t see a word.
A concrete example makes it obvious. Picture a category page whose initial HTML is basically this:
<body>
<div id="products"></div>
<script src="/app.js"></script>
</body>JavaScript fetches the products from an API and fills the grid. A person sees shoes, prices, and descriptions. GPTBot sees the empty div. No products, no prices, nothing to cite.
This isn’t a gap the AI companies are rushing to close, it’s a design choice. Fetching HTML is fast and cheap. Running JavaScript at crawl scale is slow and expensive, and AI crawlers use tight timeouts, often just a few seconds. The one real exception is Google’s Gemini, which inherits Googlebot’s rendering engine and can execute JavaScript, and Applebot renders too. But ChatGPT drives the large majority of AI referral traffic and it doesn’t render, so leaning on the exception is a thin bet. There’s a second reason to care: a large share of ChatGPT’s web answers reportedly draw on Bing’s index, and Bingbot’s own rendering is limited, so a fully client-rendered site risks being invisible from two directions at once.
Watch out: “but Google indexes our JavaScript fine” is not the same as “AI can read it.” Google spent a decade building a rendering pipeline. The AI crawlers didn’t, and they read raw HTML. Solving JavaScript for Google does not solve it for ChatGPT, Claude, or Perplexity.
4) How to diagnose the problem
You can check any page in under a minute. Three quick tests, and two that go deeper.
- View Page Source. In your browser, right-click and choose View Page Source, not Inspect. If your actual body text is in there, it’s server-rendered and AI can read it. If you see mostly an empty container like <div id=”root”> and script tags, it’s client-rendered and invisible.
- Disable JavaScript and reload. Turn JavaScript off in your browser settings and load the page. Whatever disappears is what AI crawlers can’t see.
- curl with a crawler user-agent. Run the request as the bot and search the output for your key content:
curl -A "GPTBot" https://yoursite.com/your-page | grep "text you expect to see"- Ask the AI directly. Prompt ChatGPT, Claude, or Perplexity: “Read the page at [URL] and summarize the first two paragraphs.” A generic answer or an error means it couldn’t read your content.
- Read your server logs. Filter for GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, and PerplexityBot. Visits to thin-HTML pages, or a run of 403, 429, or 404 responses, point straight at the problem. Our AI Visibility Checker automates a version of these checks, and tracking your AI visibility covers how to run them as a routine.
5) How to fix it (SSR, SSG, prerendering)
The goal isn’t to drop your framework. It’s to make sure the content AI needs is in the initial HTML. Three ways there, in rough order of preference.
- Server-side rendering (SSR). The server builds the full HTML before sending it, then the page hydrates for interactivity in the browser. Next.js (React), Nuxt (Vue), SvelteKit, and Angular Universal all support this. Best default for content that changes often.
- Static site generation (SSG). Pages are built to full HTML ahead of time. Astro, Hugo, Eleventy, and Gatsby are built for this. Ideal for stable content like blog posts and documentation, which is most of what AI cites anyway.
- Prerendering. A service such as Prerender.io detects a crawler, renders the page with a headless browser, and serves a static HTML snapshot to the bot while humans get the live app. A lighter bridge if a full migration isn’t realistic, and a fair one here because these crawlers can’t render at all.
Whichever you pick, a few rules hold. Put the content that matters, main copy, product data, headings, and internal links, into the initial HTML. Make every important page a real URL that returns full HTML, because client-side routing that swaps content without a real URL is a dead end for non-rendering bots. And keep client-side rendering for the things it’s good at: view counters, live chat, animations, and social feeds. Just not for anything you want cited. This is the same principle behind machine-first architecture.
Good sign: in one case study, AI bots jumped to nearly half of all requests soon after a single-page app added prerendering. Crawlers re-engage fast once the content is reachable, so fixes here tend to show up quickly.
For the general JavaScript-SEO treatment, server-side versus client-side rendering for Google and how its rendering pipeline works, see our technical SEO guide.
6) Other read-blockers: tabs, walls, and slow pages
Rendering is the big one, but a few other patterns hide content from crawlers that don’t click or wait.
- Content behind interactions. Tabs, accordions, “load more” buttons, infinite scroll, and modals that build their content on click won’t be seen. If the content matters, make sure it’s in the HTML even when it’s visually collapsed, or provide a static version.
- Walls and interstitials. Login walls, hard paywalls, and aggressive cookie-consent or age gates can block the content behind them. Decide what you want public and make sure it’s reachable without an action the bot can’t perform.
- Slow server responses. With timeouts measured in seconds, a slow time-to-first-byte can get a page abandoned before it’s read. A fast server response helps crawlers as much as it helps users.
One schema note, since it comes up: putting your structured data in the initial HTML is good practice, but schema won’t rescue content that isn’t there, and as our schema guide covers, AI engines read your visible text at retrieval anyway. Fix the rendered content first.
Sources used for this guide
Because rendering claims get muddled with Google’s behavior, this guide leans on primary crawler analysis and the vendors’ own docs.
| Source | What it supports |
|---|---|
| Vercel & MERJ, “The rise of the AI crawler” | 500M+ GPTBot fetches with zero JavaScript execution; the JS-file fetch rates; none of the major AI crawlers render JS. |
| Crawler-behavior analyses (2026) | Crawler-by-crawler rendering breakdown; Gemini and Applebot as the exceptions; the split-visibility problem and tight timeouts. |
| OpenAI crawler documentation | GPTBot, OAI-SearchBot, and ChatGPT-User behavior and crawl patterns. |
| Cloudflare announcement (July 2025) | Default AI-crawler blocking on new domains, a common hidden access blocker. |
FAQ: AI crawler access and rendering
Do AI crawlers run JavaScript?
Almost none do. GPTBot, ClaudeBot, PerplexityBot, and the other major AI crawlers read the raw HTML and ignore scripts. The exceptions are Google’s Gemini, which uses Googlebot’s rendering engine, and Applebot.
Can a page rank on Google but be invisible to ChatGPT?
Yes. Google renders JavaScript and indexes the result, so a client-rendered page can rank number one. ChatGPT, Claude, and Perplexity read only the raw HTML, so the same page can be a blank shell to them.
How do I check if AI can read my page?
Use View Page Source and look for your body text, disable JavaScript and reload to see what vanishes, or run curl with a GPTBot user-agent and search for your key content. You can also ask an AI to summarize the page and see if it returns your actual content.
What’s the fix for a JavaScript-heavy site?
Serve your critical content in the initial HTML using server-side rendering (Next.js, Nuxt, SvelteKit), static generation (Astro, Hugo, Gatsby), or a prerendering service. Keep client-side rendering only for non-essential enhancements.
Why isn’t my site in ChatGPT even though robots.txt allows it?
Usually a firewall or CDN returning 429, an error response like a 404, or client-rendered content the bot can’t read. Check your server logs for the AI user-agents and the status codes they get. Once access is fixed, getting recommended by ChatGPT is the next step.
Does Cloudflare block AI crawlers?
It began blocking AI crawlers by default on new domains in July 2025, so if your site is on Cloudflare, check your bot settings to confirm the crawlers you want are allowed.
Is client-side rendering always bad for AI?
No. It’s fine for things like counters, chat widgets, and animations. It’s a problem only when the content you want cited exists solely after JavaScript runs.
Should I verify AI bots by IP?
It’s good practice, since crawlers can be spoofed, but make sure your rules don’t block the real ones. Match requests against the vendors’ published IP ranges or use reverse DNS.
Conclusion: reach and read are the price of entry
Reaching and reading are what make a page eligible to be cited, and both are easy to get wrong without noticing. Make sure nothing at your firewall or CDN is quietly blocking the bots, and make sure your content lives in the initial HTML, not behind JavaScript a crawler won’t run. Do those two things and your pages can finally be seen. The content and consensus work does the rest.
With access and rendering sorted, the rest of this cluster is about what you put in that reachable HTML. Revisit the complete AI crawler setup and the best schema for AI citations, then move on to writing answer-first content AI will quote, all part of our AI search optimization guide.
How we do it: At Search Counsel Co. we start every AI-visibility engagement by confirming crawlers can reach and read the site, through our [FRAMEWORK NAME] process, because there’s no point optimizing content a machine can’t see. If you’d rather hand it off, see our AI SEO and GEO services.
Editorial note: This guide is for general marketing and technical education. Crawler rendering behavior and infrastructure defaults change quickly, so verify against the current crawler documentation and your own server logs before you rely on any single point.