Discussion
  • Crawling
  • Technical SEO

After a six-hour outage, how quickly did crawl rate and rankings recover for you, and did serving 503 rather than 500 make any visible difference?

Published 1 reply

Discussion starter: a technical SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.

Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.

1 reply

  • admin
    Admin
    Original poster

    After an outage of a few hours, the usual pattern is a dip in Googlebot’s crawl rate during and shortly after the incident, recovery over the following days as the server responds normally, and little or no lasting effect on rankings. Google’s documentation says that server errors cause its crawlers to slow down temporarily, that already indexed URLs are kept for a while, and that URLs returning errors persistently are eventually dropped. A six-hour outage is well short of “persistently”. Serving 503 rather than 500 is the correct signal for planned or temporary unavailability and is good practice, but for a short outage the practical difference in how Google treats the two is small, because both are server errors. What makes a real difference is not serving an error page with a 200 status, and not letting robots.txt fail.

    What Google does with server errors

    Response during the outage Likely effect
    503 or 500 on pages Crawling slows; indexed pages kept; recovery as the site responds normally again
    Error page returned with status 200 Google may index the error content in place of real pages, or treat pages as soft 404s
    Timeouts or no response Similar to server errors; crawl rate reduced
    robots.txt returning server errors Google may stop crawling the whole site until robots.txt is available again
    CDN serving cached pages while the origin is down Often no visible effect for crawlers

    How recovery usually looks

    1. During the outage: Crawl stats show errors and falling request counts; host status may show problems.
    2. First days after: crawling resumes, often cautiously, then climbs as responses are consistently fast and successful.
    3. Within a week or two: crawl rate back to normal for most sites; rankings typically unaffected, or brief fluctuations that settle.

    Longer outages, repeated incidents, or outages during which pages served misleading content take longer to recover from, and may cause some pages to drop from the index until recrawled.

    Why 503 is still the right choice

    A 503 tells every client, including crawlers, monitoring tools and caches, that the problem is temporary. Paired with a Retry-After header during planned maintenance, it communicates intent clearly. A 500 says something went wrong unexpectedly. For Google, both mean “slow down and come back later”; for your own monitoring and for other clients, the distinction is useful. Use 503 for maintenance and controlled degradation, and let genuine failures return whatever the failing system returns, while making sure it is a 5xx and not a 200.

    Precautions that matter more than the status code

    • Maintenance pages must not return 200. A friendly “we’ll be back soon” page with a success status invites Google to index it in place of your content.
    • Keep robots.txt available. Serve it from the CDN or a static host independent of the application, so an application outage does not make it fail.
    • Use the CDN’s stale content options, where appropriate, to serve cached pages when the origin errors.
    • Avoid redirecting everything to a status page, which Google may treat as a redirect of every URL to one destination.
    • Monitor from outside, checking status codes on key URLs and robots.txt every few minutes.

    Hypothetical example:

    An ecommerce site’s database fails for six hours on a weekday. The application returns 500 errors on product pages; robots.txt, served from the CDN, stays available. Crawl stats show a sharp dip in successful requests that day and a gradual return to normal over four days. Search clicks fall during the outage, as users cannot reach the site, and recover the next day. Rankings for tracked queries show no meaningful change. A month earlier, a competitor’s outage had served a maintenance page with a 200 status; some of its product pages briefly appeared in results with “We’ll be back soon” as the snippet.

    Watch for this:

    If robots.txt returns server errors for an extended period, Google may pause crawling of the site entirely. That turns a partial outage into a full crawling stop. Robots.txt availability deserves its own monitoring and hosting arrangement.

    After the incident

    Check Crawl stats for host status and response codes, confirm that key pages return 200 and correct content, review the Page indexing report over the following weeks for unusual increases in server errors or soft 404s, and look at search clicks by template. If everything returns to normal within a week or two, no further action is needed. If some pages dropped out, request indexing for the most important ones and make sure sitemaps are current.

    A short outage is rarely a ranking event. Return proper 5xx codes, use 503 for planned maintenance, never serve error pages with a 200, keep robots.txt independently available, and let crawling recover on its own over the following days.

    Need help with this on your own site?

    Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.

    See SEO services

Add a reply

Replies are open to approved members. Log in or apply to join.