Cut crawl errors with a practical Google Search Console check

Hero illustration: a conceptual website investigation, not a Search Console screenshot or measured results.

If you want to reduce crawl errors in Google Search Console, start by separating genuine access problems from pages that are simply not meant to be indexed. Then work through the errors in order of impact, beginning with important pages that should be reachable, indexable and linked properly.

That sounds simple, but most messy reports are not caused by one dramatic technical fault. They are usually a mix of old URLs, template changes, redirects, deleted content, blocked sections and internal links that were never cleaned up. The useful check is not “how many errors do we have?” but “which patterns stop Googlebot reaching pages that matter, and why?”

What counts as a crawl problem and what does not

In Google Search Console, people often lump several different reports together and call all of them “crawl errors”. That creates confusion, because not every excluded page is a problem, and not every non-indexed page means Google could not crawl it.

A practical distinction helps:

1. Crawl problems are cases where Googlebot tried to fetch a URL and hit a technical obstacle or an unhelpful response.
2. Indexing exclusions are cases where Google could access the URL, but decided not to index it, or was told not to.
3. Normal non-indexed pages are URLs that exist for users or systems, but are not supposed to appear in search.

Google’s own documentation for the Page indexing report makes this point clearly. An excluded URL is not automatically an error, and the reason matters (Google Page indexing report).

Examples of actual crawl problems include:

• Server errors, usually 5xx responses
• Redirect loops
• A destination that times out repeatedly
• A robots.txt block on a URL you actually want crawled
• Broken internal links that send Googlebot to dead pages

Examples of indexing exclusions that may be perfectly normal include:

• Alternate pages with canonical tags
• Pages marked noindex
• Duplicates that Google chose not to index
• Parameter URLs you do not want in search results
• Admin, basket, login or internal search pages

Examples of normal non-indexed pages include:

• Thank-you pages after form submission
• Filter combinations with no standalone value
• Staging remnants, if inaccessible to the public and not linked
• Printer-friendly duplicates
• Tag archives on small sites where they add no search value

This matters because the fix is different. If a page is excluded because it is a duplicate with a valid canonical, that is not the same as a server error on your main service page. One may need no action at all. The other may be blocking discovery and re-crawling.

For beginners, this is often the biggest step in understanding how to reduce crawl errors in Google Search Console. You do not reduce them by forcing every URL into the index. You reduce them by making important URLs consistently accessible, and by making low-value URLs less noisy.

If you need a broader grounding first, our starting plan for SEO beginners and SEO basics guide set out the main concepts without tool hype.

How to find the errors worth fixing first

Open Google Search Console and go to Indexing, then Page indexing. Do not start by chasing every row in the report. Start by sorting for impact.

A useful order is:

1. Pages that should rank or support ranking
2. Pages that receive internal links from prominent sections
3. Patterns affecting many URLs
4. Low-value or obsolete URLs last

In the Page indexing report, click each reason and inspect examples. You are looking for patterns, not one-off oddities. Ask:

• Is this a key page, or a page type?
• Is the issue sitewide, section-wide or isolated?
• Does the URL still need to exist?
• Is the reported reason consistent with what a live request returns now?

Then use URL Inspection on representative examples. The indexed result describes the version Google last processed, including its selected canonical. The live test checks current fetchability and some indexing conditions; it does not show what canonical Google will select or guarantee future indexing.

A sensible triage method is:

1. Check business-critical URLs first

Look at your homepage, core service pages, category pages, main guides, and any pages that attract links or enquiries. If these are affected by server errors, redirect problems or accidental blocking, fix them before you touch anything else.

2. Group by pattern

Ten broken blog URLs from a retired folder usually matter less than one template bug that injects bad internal links across the whole site. If the same issue appears on many URLs with the same path pattern, treat it as one root cause until proven otherwise.

3. Compare report labels with reality

Search Console reports are not always real-time. A URL listed under an error reason may already be fixed, or may now show a different issue. Use live inspection and, if needed, a browser extension, server log sample or command-line header check to confirm the current response.

4. Ignore noise that does not affect search value

If old campaign URLs, attachment pages or expired tag pages are excluded and unimportant, do not let them distract you from crawl access to pages that matter.

This evidence-led approach is worth keeping in mind with any Search Console workflow. We covered the strengths and limits of the tool in more detail in our piece on what Search Console evidence can and cannot tell you.

The main causes behind recurring crawl errors

Most recurring crawl issues fall into a small number of buckets.

Server errors

A 5xx response means the problem is at the server or application layer. Common examples are 500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable and 504 Gateway Timeout.

Typical causes include:

• Hosting instability
• PHP or plugin failures
• Database timeouts
• Security tools blocking legitimate crawler requests
• Resource limits during crawl bursts

A short-lived 503 during maintenance is not the same as repeated 500s on important pages. Repeated 5xx responses deserve urgent attention because Googlebot cannot reliably fetch the content.

Redirect loops and bad chains

Redirects are normal during site changes, but they become crawl waste when:

• URL A redirects to B, then back to A
• A long chain sends Googlebot through several hops
• Internal links still point to redirected URLs instead of the final destination
• HTTP and HTTPS or www and non-www rules conflict

Loops usually stop access altogether. Chains may still resolve, but they are inefficient and easy to break later.

Soft 404s

A soft 404 can occur when Google interprets the response as missing content or an error page even though the server returns a successful status. Common examples are thin empty category pages, “product not found” templates returning 200 OK, or search pages with no meaningful content.

Google can classify these as soft 404s even though the server says the page exists. That is not a crawl blockage in the strictest sense, but it often points to weak URL handling and poor internal linking.

Blocked URLs

A blocked URL may be blocked by robots.txt, or effectively blocked by login walls, permissions or scripts that prevent useful fetching.

Robots.txt is often misunderstood. Blocking a URL in robots.txt stops compliant crawlers from fetching it, but does not itself remove the URL from search if Google learns about it elsewhere. Google documents how discovery and crawling work in its starter guidance (Google SEO Starter Guide).

OpenAI documents that OAI-SearchBot search access is separate from GPTBot training controls (OpenAI crawler documentation). Those controls are not a fix for Google Search Console crawl errors.

Broken internal links

These are among the most fixable causes. If your own navigation, body links, breadcrumbs, related posts or XML sitemap keep pointing to dead URLs, you are repeatedly asking Googlebot to crawl things that no longer work.

Broken internal links often appear after:

• Slug changes without link updates
• Content deletions
• CMS migrations
• Category merges
• Manual copy-and-paste linking in old articles

A step-by-step fix process for each error type

The best diagnostic order is the same for most issues. Confirm the live state first, then trace the cause, then apply the smallest correct fix.

1. Test the live URL

In URL Inspection, use the live test where available. Also open the URL in a browser and check the HTTP response if you can. You want to know:

• Does it load?
• What status code does it return?
• Is there a redirect?
• Is the content actually there?
• Is it blocked from crawling or indexing?

Do not rely only on the report label.

2. Decide whether the URL should exist and be indexed

Before fixing anything technical, ask:

• Is this page still needed?
• Should users be able to access it?
• Should search engines index it?

If the answer is no, the correct fix may be removal, a 410, a noindex, or simply leaving it excluded. If the answer is yes, continue.

3. Check internal links and sitemap inclusion

If a URL should not exist, remove internal links to it and remove it from the XML sitemap. If it should exist, make sure internal links point directly to the correct canonical URL.

This step matters because a sitemap is a hint, not a guarantee, but it should not keep advertising broken URLs to Googlebot.

4. Apply the right fix by error type

For server errors

• Reproduce the issue on the live URL
• Check whether it affects one page or a pattern
• Review hosting logs or application error logs
• Test recent changes in a staging environment and use a controlled rollback if they caused the problem
• Check firewall, rate limiting or bot protection settings
• If maintenance is temporary, use 503 appropriately and restore service promptly

After the fix, request reindexing only for important URLs. The main goal is that future crawls succeed consistently.

For redirect loops or chains

• Map the full redirect path
• Identify conflicting rules in CMS settings, server config or plugins
• Update internal links to the final destination
• Replace chains with a single hop where possible
• Update legacy redirect destinations to the final equivalent URL, preserving access from old addresses

If a page was permanently replaced, a single 301 to the best equivalent page is usually cleaner than layered historical rules.

For soft 404s

• Decide whether the page should exist
• If not, return a real 404 or 410
• If yes, improve the page so it has distinct, useful content
• Remove empty templates and placeholder copy
• Avoid returning 200 OK for “not found” messages

A genuine out-of-stock or low-stock page is not automatically a soft 404. The question is whether the page still serves a clear purpose for users.

For blocked URLs

• Check robots.txt rules carefully
• Confirm whether the blocked URL should be crawlable at all
• If it should be indexed, remove the blocking rule and allow access
• If it should stay out of search, consider whether noindex on an accessible page is more appropriate than a crawl block

Remember the difference between crawling and indexing. A robots.txt block can stop Googlebot seeing a noindex tag on the page itself.

For broken internal links

• Find every internal source linking to the broken URL
• Update links to the correct live destination
• Restore the page if it was removed by mistake
• Return 404 or 410 if the old page is gone for good and has no equivalent
• Add a redirect only where there is a close replacement

Do not redirect every missing page to the homepage. That usually creates poor user experience and can look like a soft 404 pattern.

5. Validate at pattern level, not only one URL

Once one example is fixed, test several others from the same issue group. It is common to fix one page manually while the template, plugin or export process continues generating the same problem elsewhere.

How to reduce future errors through site housekeeping

The best way to reduce crawl errors over time is to make your site less untidy.

Keep internal linking clean

Internal links are one of the strongest signals of what you consider important. Descriptive links and sensible navigation also help discovery, which Google’s starter guidance explicitly supports (Google SEO Starter Guide).

Practical habits:

• Update links when slugs change
• Avoid linking to redirected URLs in new content
• Review footer, menu and related-post modules after site changes
• Fix broken links in old evergreen articles

Maintain a realistic XML sitemap

Your sitemap should list canonical URLs you want crawled and indexed. It should not be a dumping ground for every URL the CMS can generate.

Remove:

• 404 pages
• Redirected URLs
• Noindex pages
• Duplicate parameter variants
• Obsolete attachments or archives with no search purpose

Use status codes deliberately

Return the right response for the right situation:

• 200 for a live page
• 301 for a permanent move
• 404 for a missing page
• 410 when content is intentionally gone and not coming back
• 503 for temporary maintenance

Many recurring Search Console problems start with sites returning 200 for everything.

Prune low-value content carefully

Content pruning is not about deleting pages because they have low traffic. It is about reducing clutter that creates duplicate, empty or outdated URLs.

Good pruning questions include:

• Does this page answer a real need?
• Is it distinct from other pages?
• Does it still earn internal prominence?
• Is there a better URL that should replace it?

If you remove content, tidy the internal links and sitemap at the same time. Otherwise you create fresh crawl noise.

Build a monitoring habit

A monthly check is enough for many small and medium UK sites. Larger publishers or ecommerce sites may need more frequent review.

A practical routine is:

• Check Page indexing for new patterns
• Review Crawl stats if available
• Inspect a sample of key URLs after major site changes
• Check XML sitemap health
• Spot-check internal links in recently edited content

If you are also trying to understand how search evidence differs from AI visibility evidence, our AI search and visibility guide explains why a crawler visit, an index entry, a citation and a click are different things. Google’s own guidance also states that existing search fundamentals apply to its AI features, and no special AI schema or text file is required (Google AI features).

Crawl reports become manageable when you stop treating every exclusion as an emergency. Focus on important pages, confirm the live response, fix the root pattern, and clean up the signals your own site sends. That is the practical route to reducing crawl errors in Google Search Console, without turning routine maintenance into a panic.