Technical SEO Guide
How to Find and Fix Crawl Errors in Google Search Console
Crawl errors are the quiet reason good pages never show up. When a crawler can’t fetch a page, that page can’t rank in Google or get cited in an AI answer. This guide shows you exactly where to find crawl errors now that the old report is gone, and how to fix each type.
By Rahul Saini, Author at Search Counsel Co. Last updated [July] 2026.
Featured answer: what are crawl errors?
A crawl error happens when a search engine tries to fetch a page and can’t retrieve it properly. Crawl errors split into two groups: site errors that can block your whole site, and URL errors that affect single pages. Left unfixed, they keep pages out of search results and out of AI answers.
Looking for the “Crawl Errors” report? It’s gone. Google retired the standalone Crawl Errors report in 2019. That data now lives in three places inside Search Console: the Pages report, the Crawl Stats report, and the URL Inspection tool. This guide uses all three.
Two Types
Site vs URL
Site errors block everything. URL errors hit single pages.
Where to Look
Three reports
Pages, Crawl Stats, and URL Inspection, all free in Search Console.
Fix Order
Big first
Site-wide problems, then errors on your highest-traffic pages.
AI Stakes
Not just Google
A page that won’t load for Googlebot won’t load for AI crawlers either.
Jump to what you need
Article note: Written by Rahul Saini at Search Counsel Co. Checked against Google’s crawling documentation and current Search Console behavior. Search Console’s interface changes often, so confirm report names against the live tool if anything looks different.
1) What crawl errors are (and why they matter)
Crawling is how a search engine discovers your pages. Googlebot follows links, requests each URL from your server, and reads what comes back. A crawl error is any point where that fetch fails, so Google never gets a usable copy of the page. If Google can’t fetch a page, it can’t index it, and a page that isn’t indexed can’t rank.
Errors sort into two groups, and the split decides how urgent each one is.
- Site errors affect your whole site and are the ones to panic about. They include DNS failures (crawlers can’t resolve your domain), server connectivity problems (your server times out or returns 5xx errors across the board), and a robots.txt that can’t be fetched. Any of these can stop crawling site-wide.
- URL errors affect individual pages: a broken link here, a bad redirect there, a page that returns the wrong status code. One is minor. Thousands, or a handful on important pages, add up.
This split is the same mental model Google has always used, even though the report that once carried the name is gone. Fix the site-wide problems first, then work through the page-level ones. For the bigger picture of how crawling fits with rendering and indexing, see our complete guide to technical SEO.
2) How to find crawl errors in Search Console
Google Search Console is free and it’s where you start. Since the old Crawl Errors report was retired, the data is spread across three tools. Here’s how to work them in order.
Step 1: Open the Pages report. In the left menu, go to Indexing, then Pages. This report (renamed from the old Coverage report) shows how many pages are indexed and how many aren’t. Scroll to the section titled “Why pages aren’t indexed” and click any reason to see the exact URLs affected. Export that list so you can group and track it.
Step 2: Check the Crawl Stats report for site-wide trouble. Go to Settings, then Crawl stats (this is available for domain or root-level properties). Look at Host status first, because that’s where site errors show up: DNS problems, server connectivity, and robots.txt fetch failures. A red or amber flag here is your most urgent signal. The response breakdown also tells you what share of requests returned 5xx server errors, which is a direct sign your hosting is struggling.
Step 3: Inspect specific URLs. Paste any problem URL into the URL Inspection tool at the top of Search Console. It tells you whether the page is indexed, what Googlebot last saw, and whether anything is blocking it. Use Test Live URL to run a fresh crawl and see the rendered page and any blocked resources, which is how you diagnose soft 404s and rendering failures.
From experience: Search Console only shows errors for URLs Google has actually tried to crawl, and it can lag on new or recently changed pages. When you need a complete, current picture, crawl the site yourself (more on that below). The two views together catch far more than either one alone.
3) Every crawl error type, and how to fix it
Each status in the Pages report points to a different problem with a different fix. Here’s the reference table. Match the label Search Console shows you to the row, then apply the fix.
| Error (as shown in Search Console) | What it means | How to fix it |
|---|---|---|
| Server error (5xx) | Googlebot reached your server but got a 500-level response. | Check hosting logs and uptime, add caching, and upgrade if errors spike with traffic. Use a 503 with a Retry-After header for planned maintenance. Treat as urgent: repeated 5xx errors slow crawling across your whole site. |
| Not found (404) | The URL doesn’t exist. | If the content moved, add a 301 redirect to the new URL. If it’s gone for good, let it 404 (or return 410) and remove it from your sitemap and internal links. Not every 404 needs fixing. |
| Soft 404 | The page loads with a 200 status but looks empty or broken to Google. | Return a real 404 or 410 if the page is gone, or add genuine content if it should exist. Often caused by blocked resources or a page that failed to render. Test it with Live URL. |
| Redirect error | Google couldn’t follow the redirect: a loop, a chain that’s too long, or a broken destination. | Fix loops and chains. Point each redirect straight at a final URL that returns 200, and update internal links to that final URL. |
| Blocked by robots.txt | Your robots.txt is stopping the crawl. | Remove the disallow rule if you want the page indexed. If you don’t want it indexed, remove it from your sitemap. Note that robots.txt blocks crawling, not indexing, so use noindex if the goal is to keep it out of results. |
| Blocked due to access forbidden (403) | Your server refused Googlebot. | Fix the firewall, WAF, or security rule if the page should be public. Keep it out of your sitemap and off public links if it should stay private. |
| Submitted URL marked noindex | The page is in your sitemap but carries a noindex tag. | Remove the noindex if the page should rank, or remove the URL from your sitemap if it shouldn’t. Your signals should agree. |
| Page with redirect / Alternate page with canonical | Google chose a different URL: the redirect target, or the canonical version. | Usually nothing to fix. Point internal links and your sitemap at the final or canonical URL so your signals are clean. |
| Discovered or crawled, currently not indexed | Google knows the URL, or has crawled it, but hasn’t indexed it. | Usually a quality or crawl-budget signal, not a hard error. Strengthen internal links to the page and improve the content. Don’t spam the request-indexing button. |
Two of those rows, discovered/crawled but not indexed and orphan-related issues, are really indexing problems more than fetch failures. They have their own playbook in our guide to fixing indexing problems.
4) How to prioritize your fixes
You’ll rarely have zero errors, and you don’t need zero. What you need is a clear order, so you spend time where it moves the needle. Work top to bottom:
- Site-wide problems first. Server errors, DNS issues, and robots.txt failures affect every page, so they’re always the priority. A robots.txt problem or a wave of 5xx errors can pull pages out of the index if it drags on.
- High-traffic page errors next. Cross-reference your error list with analytics. A 404 on a page that gets 10,000 visits a month matters far more than the same error on a page nobody reaches.
- Everything else after that. Isolated 404s on long-dead pages, or “alternate page with canonical” notices, are often fine to leave. Don’t burn hours chasing errors that aren’t costing you anything.
Once you’ve fixed an error type, click Validate Fix in Search Console. That tells Google to recrawl the affected URLs and clear them from the report if they’re resolved, which also keeps new errors from getting buried under old ones.
5) Why crawl errors also cost AI citations
Crawl errors don’t only cost you Google rankings. They cost you AI answers too, and the mechanism is simple. AI answer engines like ChatGPT, Perplexity, and Google’s AI Overviews fetch live pages through their own crawlers. A page that returns a 404 or a 5xx, or that sits behind a robots.txt block, is exactly as unreachable to OAI-SearchBot or PerplexityBot as it is to Googlebot. If the crawler can’t fetch it, it can’t quote it.
A robots.txt fetch failure is worse still, because it can stall every crawler at once, search and AI alike. And there’s a softer cost: a site riddled with errors reads as low quality and unreliable, which is the kind of source AI systems tend to skip when they choose what to cite. Clean crawl health is table stakes for showing up in AI answers, which is why it sits at the front of our AI search optimization guide and in the AI-readiness section of the technical SEO pillar.
6) Tools beyond Search Console
Search Console shows you what Googlebot has hit. To see everything else, add a crawler and your server logs.
A desktop crawler like Screaming Frog scans your site on demand and finds errors in real time, including ones Google hasn’t tried yet. More useful, it shows you the context Search Console hides: which pages link to a broken URL (the source pages), the full redirect chain behind each redirect, and orphan pages that no internal link points to. Ahrefs and Semrush run similar site audits in the cloud and flag issues in bulk, which is worth it once your site is large.
Server logs are the ground truth. They record every request a crawler actually made and the status code it got back, so they catch problems no simulated crawl will, and they show you how often each bot visits. For most sites a weekly Search Console check plus an occasional full crawl is enough. For large or fast-changing sites, log analysis earns its place.
Free tool
Run the Crawlability and AI-Readiness Checker
Enter a URL and it checks whether Googlebot and the major AI crawlers can reach and read your pages, and flags the crawl and indexing problems that keep you out of search and AI answers. Check your site. If you’d rather have it handled, our technical SEO and site audit service runs this through our [FRAMEWORK NAME] process.
7) How to prevent crawl errors
Most crawl errors are avoidable. A few habits keep them from piling up:
- Redirect every URL change. When you delete or move a page, add a 301 to the closest relevant page so links don’t break.
- Keep your sitemap clean. It should list only canonical URLs that return a 200 status. Drop anything redirected, noindexed, or gone.
- Link every page. A page with no internal links pointing to it is an orphan, and crawlers may never find it. Good internal linking is the cheapest prevention there is.
- Watch server health. Slow responses and 5xx errors under load are a crawl problem before they’re an uptime problem. Caching and adequate hosting head them off.
- Check after every launch or migration. Relaunches are where site-wide errors sneak in, a stray noindex or a copied staging robots.txt. Run the reports the day the change goes live, not a month later.
8) Sources used for this guide
This guide leans on Google’s own documentation and current Search Console behavior rather than secondhand claims, because the tool’s interface changes often.
| Source | What it supports |
|---|---|
| Google Search Central, “Troubleshoot crawling errors” | Crawl Stats and host availability, soft 404 causes, and handling blocked resources. |
| Google Search Central, Page indexing (Pages) report documentation | The “why pages aren’t indexed” statuses and what each one means. |
| Google Search Console release history | The retirement of the standalone Crawl Errors report and the move to the current reports. |
| OpenAI and Perplexity crawler documentation | That AI retrieval bots fetch live pages, so a 404, 5xx, or robots.txt block blocks them too. |
FAQ: crawl errors
Where is the crawl errors report in Google Search Console?
There isn’t a single “Crawl Errors” report anymore. Google retired it in 2019. Crawl error data now lives in the Pages report (Indexing, then Pages) for page-level issues, the Crawl Stats report (Settings) for site-level availability, and the URL Inspection tool for checking one URL at a time.
Do crawl errors hurt SEO?
They can. A page Google can’t crawl can’t be indexed or ranked, so errors on important pages cost you visibility directly. Site-wide errors like repeated 5xx responses are worse, because Google slows its crawling of your whole site and can drop pages from the index if the problem lasts.
Are 404 errors bad for SEO?
Not usually. A 404 for a page that’s truly gone is normal and expected, and a pile of 404s on dead URLs won’t sink your site. What matters is 404s on pages that should exist or that still get traffic and links. Redirect those; let the truly dead ones return 404 or 410.
What’s the difference between site errors and URL errors?
Site errors affect your entire website, so crawlers struggle to access any page. They include DNS failures, server connectivity problems, and robots.txt that can’t be fetched. URL errors affect single pages, like a 404 or a broken redirect. Fix site errors first, because they block everything downstream.
How do I fix a soft 404?
A soft 404 is a page that returns a 200 status but looks empty or broken to Google. If the page is truly gone, make it return a real 404 or 410. If it should exist, add real content and make sure its resources aren’t blocked. Use the Test Live URL feature to see what Google actually renders.
How often should I check for crawl errors?
For most sites, once a week is enough, plus a check the day after any launch, migration, or big change, since that’s when site-wide errors appear. High-traffic or frequently updated sites benefit from more frequent monitoring and occasional server-log review.
Do crawl errors affect AI search visibility?
Yes. AI answer engines fetch live pages through their own crawlers, so a page that returns a 404 or 5xx, or is blocked in robots.txt, can’t be fetched or cited by them either. Widespread errors also make a site read as low quality, which AI systems tend to avoid citing.
Conclusion: clear the errors, clear the path
Crawl errors are one of the most fixable problems in SEO, and one of the most costly to ignore. Find them in the Pages report, the Crawl Stats report, and URL Inspection. Fix the site-wide problems first, then the errors on the pages that actually earn traffic. Then keep them from coming back with clean redirects, a tidy sitemap, and a check after every change.
Once your pages are crawlable, the next questions are how efficiently Google crawls a bigger site, and why a crawlable page still might not get indexed. Those are covered in our guides to crawl budget and fixing indexing problems, and both sit under the technical SEO pillar.
Editorial note: This guide is for general marketing education. Google Search Console’s interface and report names change over time, so verify any step against the live tool, and confirm the status of a URL with URL Inspection before acting on it.