fbpx

Google’s John Mueller said on Twitter that serving too many 5xx errors, the internal server types of errors, can cause Google to slow the crawling of your web site. If you are seeing an increase in crawl errors, that is not a huge concern, but if you are serving up tons of 5xx errors that means something is significantly wrong with your web site and Google doesn’t want to pound it with crawls.
Here is the context around John’s comment:

That’s a very vague question, and a post from 2016, it makes it a bit hard to guess :). We do crawl old 404 pages from time to time, that’s nothing to worry about. It’s more problematic if they’re 5xx errors (we slow crawling), or if the URLs were meant to be indexed.
— John ☆.o(≧▽≦)o.☆ (@JohnMu) July 9, 2018
Now, this makes sense – if a site is serving up a lot of 5xx errors, trying to crawl it faster wouldn’t be prudent. So Google might slow crawling it, stop all together for a period of time and then come back.
Forum discussion at Twitter.

Call Now ButtonCall Now!
WP Twitter Auto Publish Powered By : XYZScripts.com