Nov 18th, 2017

How to Deal with Crawl Errors in Google Search Console (Google Webmaster Tools)

Almost every web developer in the world has to deal with Google’s crawl errors, and it’s better to remedy problems quickly to prevent being penalised by Google’s ranking algorithm. Crawl errors let Google know that certain elements of your website aren’t working as they should, such as broken links, and, fortunately, most issues are easier to resolve than you might expect.

Some problems, such as DNS and server errors, usually require professional assistance, though there are some things you can try first to avoid calling your web developer. However, most common problems can be fixed by just about any amateur in a matter of minutes.

If you’re trying to master the skills of a professional web developer, you’ll benefit from learning how to deal with common problems without help. First, you’ll need to familiarise yourself with Google Search Console, previously called Google Webmaster Tools, as this is usually the key to eliminating the vast majority of crawl errors.

What is Google Search Console?

Google Search Console gives you insights into your website’s performance and highlights problematic areas that could affect your search ranking, and it’s also the platform that’s most widely used to identify crawl errors. Even though Google Search Console is intuitive and easy-to-use for most amateur developers, professional developers and SEO companies also use it almost constantly.

For this article, I’ll be referring to the Crawl Errors view within Google’s Search console, which is split into two sections: URL errors and site errors. Site errors may be complex issues that require a professional’s healing touch, but URL errors are usually relatively straightforward to fix.

Before we start talking about how to refine your website maintenance skills, let’s take a look at the most common crawl errors you’re likely to come across as you optimise your website for the search engines.

List of Common Crawl Errors

  • Server error (500)
    If the Googlebot can’t navigate your website or access your URL, it will return a 500 error status code.
  • Soft 404
    Googlebot returns a “soft” 404 error code when it finds a 200 status code it thought should’ve been a 400 status code.
  • Access Denied (403)
    If one of your links leads to a URL that requires authentication, the Googlebot may return a 403 error. If you still wish to link to the URLs that are causing problems for the bot, you might want to use nofollow links, which instruct the Googlebot to disregard them.
  • Not Found (404 / 410)
    These errors return the classic “Not found” error message. Learn more about 404 and 410 status codes.
  • Moved Permanently (301)
    If the Googlebot follows a URL that redirects to a broken URL, it will return a 301 status code to the Google Search Console.
  • Other (400 / 405 / 406)
    These error codes seem to refer to problems that Google has no name for, with the error message usually being “due to an undetermined issue.” In most cases, you can fix these problems by treating them in the same way as the more common 404 errors.
  • Flash Content (Smartphone)
    This is a report that lists pages filled with flash content, which is often incompatible with smartphones. Given the rise in smartphone searches (which surpassed desktop searches in 2015), it might be wise to minimise your use of Flash.
  • Blocked (Smartphone)
    An error that highlights mobile web pages that the Googlebot could’ve accessed but didn’t have permission. It’s crucial to ensure that the bot can crawl all the pages you want indexing.

How to Eliminate URL Crawl Errors

If you’ve never checked your website for crawl errors, you’ll likely be shocked to find hundreds listed in the Google Search Console. However, although it may sound counterproductive, you need to start the process of eliminating the crawl errors by marking them all as fixed before getting to work. That way, you can keep checking for crawl errors weekly and fix them in a structured way. Plus, with any luck, the irrelevant errors won’t crop up again.

  • The 404 Error

Fortunately, the most common error status code is the easiest to fix, usually only requiring a few minutes of your time. The Googlebot reports a 404 error when it detects faulty or broken links, whether there’s an issue with a link to your URL on somebody else’s website or there are broken internal links. Here’s how you fix it:

  • Broken links from other websites:
    If another website links to a false URL, you can implement a 301 redirect to override the original URL request and instead send the user to the correct target. Alternatively, you can ask the webmaster of the linking page to make amendments, though that could be a drawn-out process.
  • Faulty sitemap entries and internal links
    Addressing this problem is as simple as correcting the false URLs through the back-end of your web hosting platform or website creation tool. It’s not a bad idea to set up 301 redirects to make sure Google removes the 404 error from its index.
  • Mystery 404s
    Sometimes, Google will give you a 404 error for an unknown reason. It may even report a 404 error on a link from a URL that doesn’t exist, in which case you can remedy the problem with a 301 redirect.

Remember to check your Google Search Console weekly to see whether or not the errors have been dealt with for good.

  • “Soft” 404 Errors

The Googlebot may return a 404 error code if it finds an empty page that it assumes is a faulty link, which means although the link led to an active URL, there was no or very little content on the page. Yet again, this issue is easily fixed by setting up 301 redirects, provided you no longer want the faulty link to lead to the URL it currently takes users to. However, if the “soft” 404 page is your target URL, you might have to add some valuable content to the page to make the Googlebot ignore it the next time it goes crawling through your website.

  • Other Crawl Errors

Unfortunately, other status codes, such as 400, 500 and 503, are not usually as straightforward to eliminate as the more common 404 errors, but it’s still vital to deal with them as quickly as possible so that Google knows your website is in good health. You can read Google’s advice to try and get rid of these errors without help, and if that fails, you might have to call your web developer.

How to Deal with Site Errors

Site errors can be difficult to eliminate, meaning your only option might be to get in touch with your web developer or hosting platform, but it’s vital to deal with site errors quickly because they prevent the Googlebot from accessing any web page on your entire site. When you check the Google Search Console, you’ll see a list of all site errors reported in the previous 90 days.

Site errors can be broken down into three categories:

  • DNS Errors
  • Server Errors
  • Website Errors

Server issues are among the most serious of crawl errors because if they persist, Google will think there are problems with your entire site and consequently penalise you in its rankings. If you have any DNS, server or website errors, it’s probably best to call your web developer to remedy the issue promptly. However, you can find out how to eliminate site crawl errors if you’d rather put your web DIY skills to the test.

Final Tips

You’ll undoubtedly feel frustrated the first time you attempt to deal with crawl errors without professional help, but with time, practice and patience, the task becomes much simpler. Keep these tips in mind for when things become extra challenging:

  • Install a WordPress plugin that detects and highlights broken links (404s).
  • Refer to Google’s handy crawl error guide.
  • Remember that relaunched websites will likely have lots of crawl errors.
  • Mark all errors as fixed before attempting to actually fix any errors if you’ve never done it before.
  • If you can’t eliminate a problem, get in touch with your web developer or SEO company.


Contact Smart Traffic for More Guidance on Crawl Errors

Here at Smart Traffic, we know that fixing crawl errors can be both challenging and annoying, especially if you have hundreds of errors that need dealing with sooner rather than later. Fortunately, when you hire our SEO specialists, you don’t have to worry about the prospect of crawl errors affecting your search rankings. Hopefully, the information above should help you get rid of most problems without help, but if you ever need assistance, we’re only a phone call away. Contact us today on 02 8205 3133 to speak with a leading SEO company in Sydney, Australia.

About the author

Michael Holder

A specialist in Google Ads, and with a wide range of SEO skills and experience, Michael has a particular interest in Local SEO strategies for small to medium sized businesses. When Michael is not working hard increasing conversions for clients Paid Advertising campaigns he enjoys traveling, dining out and watching Formula 1 and football.