By Rahul Saini, Author at Search Counsel Co. Last updated [July] 2026.

Here’s the good news if you run a newer or smaller site: technical SEO is one of the most winnable parts of search. Most of the work is specific, fixable problems with narrow intent, the kind big brands routinely neglect. A clean, fast, well-structured site can get crawled, indexed, and cited more reliably than a bloated one with far more authority. This page is the map. It explains the whole system, then points you to the deeper guide for each piece.

1) What is technical SEO?

Technical SEO is the practice of optimizing your website’s infrastructure so search engines and AI systems can crawl, render, index, and use your content. It’s the layer beneath your writing and your links. You can publish the best page on a topic, but if a crawler can’t reach it, can’t read it, or can’t tell which version to keep, none of that quality reaches a searcher or an AI answer.

People usually split SEO into three parts: on-page (the content and its structure), off-page (links and mentions that build authority), and technical (the machine-facing plumbing). Of the three, technical SEO is the only one that decides whether your pages are eligible to appear at all. The other two raise how well you compete once you’re in the running. Technical work gets you into the running.

A quick boundary, because the term gets stretched. Technical SEO covers crawling, rendering, indexing, site architecture, URLs, canonicalization, speed and Core Web Vitals, security, and structured data. It doesn’t cover keyword research, writing, or link building, which belong to on-page and off-page. For how those pieces fit together, see our guide to the types of SEO.

2) Why technical SEO now serves two audiences

For twenty years, technical SEO meant one thing: make the site easy for Googlebot to crawl and index. That job hasn’t gone away. What’s changed is that a second set of machines now reads your site and makes its own decisions, and it reads differently.

When someone asks ChatGPT, Perplexity, Google’s AI Overviews, or Claude a question, the system often fetches live web pages, pulls out the passages that answer the question, and cites a small handful of sources in the answer it writes. To be one of those sources, your page has to be reachable and readable by that engine’s crawler, and clear enough that a machine can lift a self-contained fact out of it. That’s technical SEO, aimed at a new target: not a ranking in a list, but a citation inside an answer. Our companion hub covers that discipline in depth in the AI search optimization guide.

Two differences between the audiences matter more than the rest, and they shape almost every recommendation on this page.

The first is JavaScript. Google can run JavaScript. It renders pages much the way a browser does, so content that only appears after scripts execute can still be seen, indexed, and ranked. There are limits (rendering happens on a queue, and Google leans on server-rendered HTML for anything critical), but broadly, Google copes with modern JavaScript sites. Most AI crawlers do not. Independent tests of the major ones, the crawlers behind ChatGPT, Claude, and others, found they read only the raw HTML and skip client-side JavaScript entirely. If your important content and links only exist after JavaScript runs, they’re effectively invisible to AI answer engines even when Google sees them fine.

The second is control. You no longer manage one crawler. You manage a fleet. OpenAI, Anthropic, Perplexity, and Google each run separate bots for separate jobs, and you can allow or block them independently. Get that configuration wrong, usually by accident, and you can delete yourself from an entire discovery channel without noticing. More on that below in the AI-readiness section.

3) The technical SEO pipeline: crawl, render, index, understand

The clearest way to think about technical SEO is as a pipeline. A page has to clear each stage before the next one can happen. If it stalls at any stage, everything downstream fails, no matter how good the content is. Here are the four gates.

Google’s version of this pipeline ends in ranking: crawl, render, index, rank. The AI version ends in citation: crawl, render, index, then retrieve and cite. The first three stages are shared, which is the reason strong technical fundamentals pay off on both surfaces at once. The rest of this guide walks the pipeline in order, grouped into the five areas the hub covers in detail.

4) Crawling and indexing

Crawling is discovery: a bot finds a URL and fetches it. Indexing is storage: the engine decides to keep the page and file it for later. If either step fails, the page can’t rank or be cited, so this is where technical SEO starts.

The tools you use to guide crawlers are simple in theory. An XML sitemap lists the URLs you want found. Your robots.txt tells bots which paths they may fetch. Internal links carry crawlers from page to page and signal which pages matter. On bigger sites, crawl budget becomes real: Google won’t crawl every URL every day, so you want it spending that budget on pages that matter, not on faceted-filter duplicates or dead ends.

On a small site you can ignore crawl budget. On a large one, a few patterns quietly burn it, and trimming them is often the single highest-impact technical fix:

• Faceted navigation. Filter and sort combinations can spawn thousands of near-duplicate URLs from a few hundred products.
• Session IDs and tracking parameters that mint a fresh URL for the same page.
• Infinite spaces like calendars or endless “load more” paths that never resolve.
• Long redirect chains that make a bot follow three hops to reach one page.

Indexing has its own failure modes, and they’re common. A page marked with a noindex tag that shouldn’t be there. Orphan pages with no internal links pointing at them, so nothing leads a crawler to them. And the status that frustrates everyone in Search Console, “discovered, currently not indexed,” where Google knows the URL exists but hasn’t judged it worth storing. Each has a specific fix.

5) Rendering and JavaScript

Rendering is the step between fetching a page and seeing its content. Modern sites often build the page in the browser with JavaScript, which means the HTML a bot first downloads can be nearly empty until scripts run. Whether your content survives that gap depends entirely on who’s reading.

Google renders JavaScript. It fetches the raw HTML, queues the page, runs the scripts with a headless browser (its Web Rendering Service), and indexes what it sees, close to how a person would. It isn’t instant or unlimited (there’s a rendering queue, a cap of roughly 2MB on the HTML and resources, and pages that return an error status may skip rendering), so Google’s own guidance still points toward server-side rendering or pre-rendering for anything important. But for most sites, Google handles JavaScript well enough.

AI answer engines are the opposite. Most of their crawlers don’t run JavaScript at all, so they read only the raw HTML you serve. If your headline, your body copy, your internal links, or your key facts only appear after JavaScript executes, an AI crawler sees a blank or partial page and cites someone else. This is the strongest technical argument for server-side rendering in 2026: it’s the difference between being readable to AI systems and being invisible to them.

6) Site architecture and URLs

Architecture is how your pages connect. Get it right and both crawlers and readers can reach anything in a few clicks, and the relationships between pages are obvious. Get it wrong and you create orphan pages, wasted crawl budget, and duplicate content that splits your ranking signals.

The reliable pattern is a flat, organized structure where every important page sits only a few links from the homepage, tied together with a clear internal-linking model. This is the pillar-and-cluster approach: a hub page on a broad topic, supported by focused posts that link up to it and across to each other. This very site is built that way, and this page is one of the hubs.

URLs are part of the same job. Clean, readable URLs help; messy parameter strings and redirect chains hurt. Two technical points come up constantly. First, use the right redirect: a 301 for a permanent move (it passes signals to the new URL), a 302 only for temporary ones. Second, canonicalization, which tells engines the preferred version of a page when several URLs show near-identical content. Handled well, canonical tags prevent duplicate-content dilution. Handled badly, they hide the wrong pages.

If you serve more than one country or language, hreflang is the architectural piece to get right on top of this. It tells Google which language and regional version to show which searchers. The most common cause of hreflang failure is a missing return tag, where page A points to page B but B doesn’t point back. It’s fiddly, and worth a careful setup on any international site.

7) Core Web Vitals and page experience

Once a page can be crawled, rendered, and indexed, experience signals affect how well it competes. The headline set is Core Web Vitals, Google’s three measures of real-world loading and responsiveness. They aren’t the biggest ranking factor, but a slow, janky page can hold back otherwise strong content, and speed matters for readers regardless of what any algorithm does.

There are three vitals to know. Largest Contentful Paint (LCP) measures loading, and good is under 2.5 seconds. Interaction to Next Paint (INP), which replaced First Input Delay as a Core Web Vital in March 2024, measures responsiveness, and good is under 200 milliseconds. Cumulative Layout Shift (CLS) measures visual stability, how much the page jumps around as it loads, and good is under 0.1. You can see all three for real users in the Core Web Vitals report inside Google Search Console. One more number worth watching is Time to First Byte (TTFB), how quickly your server responds, because a slow TTFB drags LCP down with it.

Two more page-experience basics round this out. Mobile matters because Google indexes the mobile version of your site first, a policy called mobile-first indexing, so the mobile version is the one that counts. And HTTPS has been a lightweight ranking signal since 2014 and is now a baseline expectation: it protects users, browsers flag sites without it, and most AI systems prefer secure sources when they choose what to cite. None of this is exotic. It’s hygiene that compounds.

8) Structured data and AI-crawler readiness

This is the newest layer, and it’s where technical SEO and AI search meet. Two jobs live here: helping machines understand your content with structured data, and making sure the right crawlers can actually reach it.

Structured data, or schema markup, is code that labels what your content is: an article, a how-to, a product, a dataset, a set of FAQs. It doesn’t change what readers see. It gives machines an unambiguous description they can trust, which helps you earn rich results in Google and gives AI systems clearer signals about your pages. Original data marked up as a dataset is especially valuable, because AI answers reach for specific, sourced facts and often cite the page that supplies them.

Crawler access is the other half, and it’s changed completely. The scale is already real: in early 2025, Cloudflare reported that AI crawlers were generating more than 50 billion requests a day across its network, close to 1% of all web traffic it saw. Every major AI provider now runs separate bots for separate purposes, and you control them independently in robots.txt. There are two families to understand. Training crawlers (GPTBot, ClaudeBot, Google-Extended, and Common Crawl’s CCBot) collect content to train future models. They don’t send you traffic or cite you. Retrieval crawlers (OAI-SearchBot, Claude-SearchBot, and PerplexityBot) fetch pages to answer live questions, and they’re the ones that put you in AI answers with a citation.

The practical upshot: you can opt out of model training while staying eligible for AI citations, by blocking the training bots and allowing the retrieval bots. OpenAI’s own documentation is blunt about the cost of getting this wrong, blocking OAI-SearchBot means your site won’t appear in ChatGPT’s search answers, even though it’s run by the same company as GPTBot. Two warnings that catch people out: a CDN setting (Cloudflare’s “block AI bots” toggle is the usual culprit) can override your robots.txt and block everything, and a noindex tag blocks AI search bots too, so a page you want cited has to stay indexable. The file-by-file detail, including where llms.txt fits alongside robots.txt, is covered in the AI hub.

The bots sort into three jobs. Once you see the pattern, the whole fleet gets easy to manage:

A sensible 2026 starting point for a site that wants AI visibility without feeding model training looks like the block below. Treat it as a template, not gospel. Blocking training bots is a real choice, and some evidence suggests allowing training can build long-term familiarity that helps citations over time. Whatever you pick, validate it before it goes live.

# Search engines: allow
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# AI retrieval bots (these get you cited): allow
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

# AI training bots: block only if you don't want to train models
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# Point every bot to your sitemap
Sitemap: https://example.com/sitemap.xml

Swap in your real sitemap URL. Remember that a CDN or firewall rule can override this file entirely, so check both layers, and note that a Disallow line under the wrong user-agent can de-index a site, so test before you ship.

9) How to run a technical SEO audit

An audit is how you find which of the problems above are actually on your site. You don’t need a big stack to start. The fastest path is to check the pipeline in order, top to bottom, and fix what’s blocking the most pages first.

A practical sequence looks like this:

• Check crawlability. Open your robots.txt and confirm you aren’t blocking anything important. Confirm your CDN isn’t silently blocking crawlers on top of it.
• Check indexing. In Google Search Console, look at the Pages report for what’s indexed and, more usefully, what isn’t and why.
• Check rendering. Use the URL Inspection tool to see the page as Googlebot renders it, and view a page’s raw HTML (before JavaScript) to see what an AI crawler would get.
• Check speed and vitals. Run key pages through PageSpeed Insights and review the Core Web Vitals report for real-user data.
• Check structure and schema. Confirm your important pages are internally linked, and validate your structured data with Google’s Rich Results Test.
• Check AI access. Confirm the retrieval bots you want are allowed, and that pages you want cited aren’t set to noindex.

Free tools cover most of this: Google Search Console, PageSpeed Insights, the URL Inspection and Rich Results tests, and a crawler like the free tier of Ahrefs or Screaming Frog for a full-site view. Paid platforms such as Semrush and Ahrefs add site-wide audits that flag issues in bulk, which is worth it once your site is large enough that manual checks miss things.

10) Common technical SEO mistakes

Most technical damage comes from a short list of avoidable errors. These are the ones worth checking first, because they’re common, high-impact, and usually accidental.

• The site-wide noindex left on after launch. A staging setting copied to production can quietly de-index an entire site. It’s the classic “traffic dropped to zero after relaunch” cause.
• A robots.txt or CDN rule blocking crawlers. One stray Disallow line, or a CDN’s AI-blocking toggle, can remove you from Google or from AI answers.
• Content that only exists after JavaScript. Fine for Google, invisible to most AI crawlers. Serve critical content in the raw HTML.
• Redirect chains and loops. Multiple hops between URLs waste crawl budget and slow users. Point redirects straight at the final destination.
• Conflicting canonical signals. Different canonical tags in raw and rendered HTML, or canonicals that point at the wrong page, send engines to index URLs you didn’t choose.
• Slow, unstable pages. Poor Core Web Vitals won’t sink great content on their own, but they hold it back and cost you readers.

11) Sources used for this guide

Because technical guidance and AI-crawler behavior change quickly, this guide leans on primary documentation and named reporting rather than secondhand claims.

FAQ: technical SEO

Conclusion: fix the plumbing, then everything else works

Technical SEO is the foundation the rest of your search strategy stands on. Content earns attention and links earn authority, but neither reaches anyone if a page can’t be crawled, rendered, indexed, and understood. That was true when Google was the only machine reading your site. It’s more true now that AI answer engines read it too, and read it more strictly.

The work is winnable, especially for smaller sites, because most of it is specific and fixable. Start with the free Crawlability and AI-Readiness Checker to see where you stand, then work down the pipeline using the deeper guides linked throughout this hub. When you’re ready to connect the technical layer to the AI-visibility side, head to the AI search optimization guide.