Google explains how to manage Googlebot traffic when migrating to a CDN

Google, in a blog post published Nov 11, 2026, said CDN migrations can cause sudden Googlebot crawl spikes, and site owners should pre-warm caches and exempt verified bots from blocking rules.

11 November 2026 1 min read Source: developers.google.com

Audio summary 0:21

Headline and summary, read by a synthetic voice.

In short

Businesses that are migrating or planning to migrate their website to a CDN and are crawled by Google; the source does not limit this to a specific sector or country.
For businesses moving their website to a CDN, this means Googlebot requests can overload the origin server before the cache warms up, and blanket bot-protection or CAPTCHA settings can inadvertently block Googlebot and cause pages to drop out of Search.
Before switching to a CDN, pre-warm the cache by fetching your highest-traffic and most-crawled URLs through it at a controlled pace; configure your CDN/WAF to exempt verified bots, including Googlebot, from CAPTCHA challenges; and if origin load becomes critical, set rate-limiting rules to return 429 (Too Many Requests) or 503 (Service Unavailable) for a short window.

What changed: Google Search Central announced that when sites move to a CDN, the initially empty (cold) cache means Googlebot requests still hit the origin server directly, and combined with blanket bot-protection rules this can get crawlers blocked and pages dropped from Search; it recommends pre-warming caches, exempting verified bots from CAPTCHA challenges, and if needed using temporary 429/503 responses to rate-limit crawling.

Who is affected

Businesses that are migrating or planning to migrate their website to a CDN and are crawled by Google; the source does not limit this to a specific sector or country.

What it means for your business

For businesses moving their website to a CDN, this means Googlebot requests can overload the origin server before the cache warms up, and blanket bot-protection or CAPTCHA settings can inadvertently block Googlebot and cause pages to drop out of Search.

What to do

Before switching to a CDN, pre-warm the cache by fetching your highest-traffic and most-crawled URLs through it at a controlled pace; configure your CDN/WAF to exempt verified bots, including Googlebot, from CAPTCHA challenges; and if origin load becomes critical, set rate-limiting rules to return 429 (Too Many Requests) or 503 (Service Unavailable) for a short window.