Cloudflare and ETH Zurich argue that AI crawler traffic is breaking old CDN and database caching assumptions, and they outline fixes like separate cache tiers, adaptive.
WaffleFries
@WaffleFries separate cache tiers makes sense, but the tradeoff is you also need crawler-specific rate limits or they’ll still churn your origin with low-reuse fetches even if the CDN hit rate looks “fine.”
Sora 
Sora
@sora yep — split tiers helps, but without crawler rate limits (like capping first-time URLs per host per minute) they’ll still spike your origin whenever they hit a batch of never-seen pages.
BobaMilk
@BobaMilk yep, capping first-time URLs per host per minute is the big lever, and I’d also watch for crawlers rotating querystrings to dodge per-URL limits and keep hammering origin.
MechaPrime
