How to Find and Fix Crawl Errors in Google Search Console

By Rahul Saini, Search Counsel Co. Last updated August 26, 2026.

Where Are Crawl Errors in Google Search Console Now?

The old standalone Crawl Errors report is gone. That means there is no single modern screen containing everything marketers casually call a “crawl error.”

Instead, Google distributes the evidence across different tools because the underlying problems are different.

For most sites, start with Page Indexing and URL Inspection. Crawl Stats becomes especially useful when you suspect a broader availability, server, DNS or robots.txt problem.

Google itself describes Crawl Stats as an advanced report and says sites with fewer than roughly 1,000 pages generally should not need this level of crawl analysis.

For the wider technical process, use SearchCounselCo’s Google Search Console guide and technical SEO guide.

Crawl Error vs Indexing Status: Do Not Confuse Them

This is the distinction that prevents wasted SEO work.

Crawling is Google’s attempt to request a URL and retrieve its resources or content.

Indexing happens after Google has enough information to process the URL and decide whether it belongs in the search index.

A URL can therefore be:

• successfully crawled but not indexed;
• intentionally blocked from crawling;
• intentionally noindexed;
• redirected correctly;
• recognized as a duplicate;
• or genuinely inaccessible because of a server, network, redirect or access failure.

That is why fixing every gray row in Page Indexing is a bad target.

The correct target is:

Every important canonical URL should have the crawl and indexing state you intentionally want.

SearchCounselCo Crawl Error Triage Matrix

Use this before touching redirects, robots.txt, noindex tags or server settings.

For deeper treatment of non-indexed URLs, use How to Fix Google Indexing Problems. Keeping that intent separate prevents this guide from becoming a second indexing article.

How to Check Crawl Errors in Google Search Console

Step 1: Open Page Indexing

Open your Search Console property and go to:

Indexing → Pages

Review the reasons shown under Why pages aren’t indexed.

Do not start fixing from the largest number automatically. A large group of alternate canonical URLs can be perfectly healthy, while one 5xx error affecting a revenue page can be urgent.

Step 2: Decide whether the state is intentional

Ask three questions:

1. Should this URL exist?
2. Should Google be allowed to crawl it?
3. Should Google index this exact URL?

This prevents you from “fixing” healthy redirects, canonical alternates or deliberately removed URLs.

Step 3: Inspect an example URL

Open an affected URL in URL Inspection.

Check:

• indexing status;
• last crawl date;
• crawl allowed;
• page fetch status;
• declared and Google-selected canonical where available;
• indexing allowed status.

Step 4: Test the live URL

If the issue could have changed since Google’s last crawl, use Test Live URL.

This helps distinguish:

“Google saw this problem before”

from:

“The problem still exists now.”

Step 5: Inspect the root cause outside Search Console

Depending on the error, check:

• HTTP response;
• server logs;
• CDN logs;
• WAF/firewall rules;
• redirect configuration;
• robots.txt;
• page source;
• rendered HTML;
• internal links;
• XML sitemap.

Step 6: Fix the cause, not the Search Console label

You cannot directly edit a Search Console error away.

Fix the real system generating it.

Step 7: Test again

Confirm the URL now behaves correctly before starting validation or requesting indexing.

Which Search Console Tool Should You Use?

Current Google Search Console Statuses You Should Understand

The exact wording matters because Search Console terminology changes over time.

Server error (5xx)

Google reached the host or attempted the request but received a server-class failure.

This can include:

• 500 Internal Server Error;
• 502 Bad Gateway;
• 503 Service Unavailable;
• 504 Gateway Timeout;
• application failures;
• CDN/origin failures;
• capacity problems.

Redirect error

Google could not successfully process the redirect path. Causes can include loops, invalid destinations or problematic chains.

URL blocked by robots.txt

Google’s crawler is disallowed from requesting the URL according to the robots rules it has processed.

Remember: blocking crawling with robots.txt is not the same as giving Google a noindex directive.

URL marked ‘noindex’

Google encountered a noindex directive and did not index the page.

If that is intentional, there is nothing to fix.

Soft 404

The URL may return a successful response such as 200, but Google’s systems interpret the resulting content as effectively missing or error-like.

401, 403 and other 4xx

These indicate access/client-response problems. For a public indexable page, investigate why Googlebot receives a failure.

Not found (404)

A 404 is not automatically bad SEO.

If the page is truly gone and has no replacement, 404 or 410 can be correct.

How to Fix 5xx and 429 Crawl Errors

Server errors deserve much more attention than an isolated expected 404.

Google documents that persistent 5xx responses can reduce crawling and eventually cause affected indexed URLs to fall out of the index if the failure continues.

Google treats 429 Too Many Requests differently from normal 4xx responses. It treats 429 as a server-overload signal.

Common root causes

• origin server overloaded;
• PHP/application worker exhaustion;
• database failures;
• CDN/origin connection problems;
• reverse proxy errors;
• WAF rate limiting;
• bot protection blocking Google;
• deployment failure;
• DNS or network instability;
• maintenance configuration.

5xx troubleshooting workflow

1. Open an affected URL in URL Inspection.
2. Check Google’s last crawl time.
3. Run a live test.
4. Check the current HTTP response separately.
5. Open Crawl Stats and inspect Host status and response-code trends.
6. Check origin, CDN and WAF logs near Google’s crawl timestamp.
7. Identify whether the response was 429, 500, 502, 503 or 504.
8. Determine whether real users were also affected.
9. Verify whether any security rule selectively blocked Googlebot.
10. Fix the capacity, application, network or firewall cause.
11. Run the live test again.

Google’s current HTTP-status documentation is available at How HTTP Status Codes Affect Google’s Crawlers.

The Page Works in My Browser but Googlebot Fails. Why?

This is one of the most important crawl-error patterns to understand.

A page loading correctly for you does not prove that Googlebot receives the same response.

Check the WAF, CDN and firewall

Security systems sometimes classify automated traffic aggressively. That can create a site that looks healthy to humans but blocks legitimate crawlers.

Verify that “Googlebot” is really Googlebot

Do not whitelist every request simply because its user-agent claims to be Googlebot.

User-agents can be spoofed.

Google provides two reliable approaches:

• match the source IP against Google’s published crawler IP ranges;
• or perform reverse DNS followed by forward DNS verification.

See Google’s official Verify Requests from Google Crawlers and Fetchers documentation.

How to Fix Redirect Errors

A normal 301 or 308 is not a crawl error.

The problem is when Google cannot successfully reach the intended destination.

Common causes

• redirect loop;
• very long redirect chain;
• destination itself redirects incorrectly;
• broken protocol or hostname rule;
• bad regex redirect;
• redirect target returns 404 or 5xx;
• HTTP/HTTPS or www/non-www conflict.

Correct fix

Whenever possible:

Old URL → one direct redirect → final canonical 200 URL

Then update internal links so users and crawlers no longer need to travel through the old URL.

For the implementation details, use SearchCounselCo’s URL structure and redirects guide.

404 Errors: Which Ones Should You Actually Fix?

This is where many crawl-error cleanups waste the most time.

Google does not expect a website to have zero 404 URLs.

A deleted URL with no replacement is supposed to become unavailable.

How to Fix Soft 404 Errors

A soft 404 is usually a mismatch between the HTTP response and what the rendered page actually communicates.

For example:

The server says:

200 OK

but the page effectively says:

“Nothing here.”

Common causes

• empty category or search-result pages;
• deleted product page still returning 200;
• database failure producing an empty template;
• missing server-side content;
• JavaScript failing to render the real page;
• API failure;
• generic “not found” message returned with 200.

If the page is really gone

Return a proper 404 or 410.

If the page should exist

Restore meaningful content and make sure Google can render it.

Use:

URL Inspection → Test Live URL → View Tested Page

Then examine the rendered output rather than assuming the browser version is what Google receives.

Robots.txt Problems: Blocked vs Unreachable

These are two different problems.

URL blocked by robots.txt

Google can retrieve robots.txt and sees a rule preventing the URL from being crawled.

If the block is intentional, that may be correct.

If you need the page crawled, change the relevant rule.

For detailed configuration, use SearchCounselCo’s robots.txt guide.

Robots.txt itself is unavailable

This is more serious.

If Google cannot reliably retrieve robots.txt because of a server, DNS or networking problem, crawling behavior changes because Google cannot confidently determine the site’s crawl rules.

Google’s current robots documentation explains that server failures while fetching robots.txt can temporarily stop or alter crawling while Google retries or uses a previously cached valid version.

See Google’s robots.txt specification documentation.

How to Fix 401, 403 and Other 4xx Crawl Problems

401 Unauthorized

A public page that should rank should not require Googlebot to authenticate.

Check:

• basic authentication;
• staging protection accidentally left enabled;
• membership/login rules;
• CDN access controls.

403 Forbidden

For an intended public page, a 403 often points toward:

• WAF rule;
• IP restriction;
• bot-management system;
• security plugin;
• server permission rule.

Before allowing a suspicious “Googlebot” request, verify that it is really from Google.

Other 4xx responses

Investigate why the page is returning a client-error response and whether the URL is supposed to be publicly accessible.

Why Search Console Still Shows an Error After You Fixed It

This is one of the most common sources of unnecessary SEO work.

Page Indexing can reflect what Google observed during an earlier crawl.

You may therefore see:

• a Page Indexing error;
• but a successful live URL test;
• because the underlying problem was corrected after Google’s recorded crawl.

Always compare the error with the Last crawl date before changing the page again.

Validate Fix vs Request Indexing vs Test Live URL

These buttons do different jobs.

Google says it can discover corrected issue instances through normal crawling even if you never start validation.

When you do start validation, let the cycle run. Google says validation commonly takes up to around two weeks, although some cases take longer.

A Passing Live URL Test Does Not Prove Every GSC Issue Is Fixed

This is an important limitation that many crawl-error guides miss.

Google says the live URL Inspection test does not test every condition represented in Page Indexing.

Most notably, live inspection does not reproduce all of Google’s:

• duplicate evaluation;
• canonical selection;
• indexing-system decisions.

So this result:

“Live test passed”

does not automatically mean:

“Google will index this URL and choose it as canonical.”

A successful fetch restores technical eligibility. Indexing and ranking remain separate decisions.

Why Search Console Does Not Show Every Affected Crawl URL

Search Console is not intended to be a complete site crawler export.

Google states that the Page Indexing report’s example list is limited to 1,000 URLs and is not guaranteed to show every affected URL.

That is one reason larger sites should combine GSC with a crawler, sitemap segmentation, CMS exports or server logs.

Why old issues can remain visible

Google also documents an issue lifetime.

An issue can remain represented until 90 days after the last known instance is gone. That does not necessarily mean an affected URL is still broken today.

Google Search Console vs SEO Crawler vs Server Logs

A strong technical diagnosis often uses all three.

When Should You Use the Crawl Stats Report?

Crawl Stats becomes especially useful when the issue appears larger than one URL.

Open:

Settings → Crawl Stats

Use it when you suspect:

• widespread 5xx responses;
• 429/rate limiting;
• DNS failures;
• host connectivity problems;
• robots.txt availability problems;
• abnormal Googlebot request patterns;
• large-site crawling inefficiency.

Start with Host status

Google groups availability signals around:

• robots.txt availability;
• DNS resolution;
• host connectivity.

Then inspect crawl responses

Look for unexpected increases in:

• 5xx;
• 429;
• 404;
• redirects;
• response time.

For true request-level analysis, use your server or CDN logs.

If your real problem is crawl efficiency across a large or rapidly changing site rather than an error, move to SearchCounselCo’s crawl budget guide.

Crawled or Discovered but Not Indexed Is a Different Workflow

Two common Search Console statuses create significant confusion.

Crawled – currently not indexed

Google says it crawled the page but has not indexed it. Google may or may not index it later, and Google specifically says there is no need to keep resubmitting the URL for crawling.

This is not a crawl fetch failure.

Discovered – currently not indexed

Google knows about the URL but has not crawled it yet.

Google says this commonly occurs when it expects crawling at that time could overload the site, so crawling is rescheduled.

On larger sites, also investigate:

• server capacity;
• URL inventory quality;
• crawl demand;
• internal linking;
• duplicate/faceted URLs;
• sitemap quality.

Do not turn this crawl-error article into a duplicate indexing guide. Continue with How to Fix Indexing Problems in Google.

Does Fixing a Crawl Error Guarantee Indexing or Rankings?

No.

Fixing a crawl failure restores an important technical prerequisite: Google can access and process the page again.

But a successful 200 response does not guarantee Google will index the URL, and indexing does not guarantee high rankings.

Google still evaluates factors such as:

• content usefulness;
• canonicalization;
• duplication;
• search demand;
• page/site signals;
• internal and external relationships;
• overall ranking competition.

The correct sequence is:

Crawlable → processable → index eligible → indexed → competitive enough to rank.

What About AI Crawlers?

A Google crawl test only tells you about Google’s ability to access the URL.

It does not prove that every other crawler has identical access.

Different crawler identities can encounter different:

• robots.txt groups;
• WAF rules;
• IP allowlists;
• rate limits;
• CDN bot-management policies.

If AI-search accessibility is important, evaluate the relevant crawler separately rather than assuming Googlebot access proves universal crawler access.

For that broader workflow, see SearchCounselCo’s AI Search Optimization guide.

How to Prevent Crawl Errors

1. Redirect URL changes properly

When important content moves, create a relevant permanent redirect and update internal links.

2. Keep XML sitemaps clean

Prioritize canonical, indexable URLs that return the intended successful response.

Remove URLs that are:

• 404;
• redirected;
• intentionally noindexed;
• non-canonical.

3. Fix broken internal links

Do not make Google repeatedly discover important pages through broken or redirected paths.

Use the internal linking guide for the wider architecture.

4. Monitor hosting and server health

Watch:

• 5xx rate;
• 429 rate;
• origin response time;
• CDN errors;
• application errors;
• database failures.

5. Review WAF and bot-management rules

Security should block abusive traffic without unintentionally blocking legitimate search crawlers.

6. Crawl after major changes

Run a technical check after:

• site migration;
• CMS migration;
• domain change;
• HTTP → HTTPS transition;
• large redirect deployment;
• theme/template launch;
• robots.txt change;
• CDN/WAF migration.

7. Check Search Console after deployment

A staging noindex or robots rule copied into production can cause much more damage than one isolated 404.

SearchCounselCo Crawl Error Triage Sheet

For a larger cleanup, do not manage crawl issues from memory.

Create a working sheet with these columns:

This turns “fix crawl errors” from an uncontrolled cleanup into a repeatable diagnostic process.

Frequently Asked Questions About Crawl Errors

Primary Sources Used for This Guide

This guide prioritizes Google’s current documentation because Search Console terminology and crawler behavior can change over time.

• Google Search Console — Page Indexing Report — current status labels, 404 guidance, noindex, soft 404, 401/403, validation, report examples and live-test limitations.
• Google Search Console — Crawl Stats Report — host status, crawl responses, DNS, server connectivity, robots.txt availability and sample URL limitations.
• Google — How HTTP Status Codes Affect Google’s Crawlers — 2xx, 4xx, 429 and 5xx handling.
• Google Search Central — Troubleshoot Crawling Errors — availability problems, crawling efficiency, server capacity and soft 404 guidance.
• Google — robots.txt Specification — handling robots.txt response codes and unavailable robots files.
• Google — Verify Requests from Google Crawlers and Fetchers — Googlebot IP and DNS verification.
• Google Search Central — Googlebot — Googlebot behavior and crawler verification guidance.
• Google — Debug Network and DNS Errors — network failures, DNS failures and crawling effects.

Final Takeaway: Fix the Root Cause, Not the Row in Search Console

A good crawl-error audit does not aim to turn every Search Console status green.

It answers four questions:

1. Should this URL exist?
2. Should Google crawl it?
3. Should Google index this exact URL?
4. Does the current technical response match that goal?

If an important URL should be available but returns a server error, redirect loop, access failure or broken response, fix it quickly.

If a deleted page correctly returns 404, an alternate URL correctly points to its canonical, or a deliberately excluded page carries noindex, do not manufacture unnecessary work.

And when Search Console still shows an error after you have made the repair, check Google’s last crawl date and run a live test before rewriting the page or changing the server again.

Once crawl access is healthy, continue with how to fix indexing problems if Google can crawl the page but still will not index it, or use the crawl budget guide when the real problem is crawling efficiency across a large or rapidly changing site.