AWS CloudFront Outage: How One Frankfurt AZ Broke Websites Worldwide

At 01:45 PDT on July 16, a failure in a packet processing subsystem inside a single Amazon Web Services availability zone in Frankfurt began propagating through CloudFront's global delivery network. By the time most of North America woke up, websites and applications across multiple continents were returning 504 timeouts and 5xx errors. AWS confirmed the root cause within hours: the affected layer is the one responsible for routing CloudFront requests to resources inside customer VPCs — a feature called VPC Origins.

What VPC Origins Does, and Why It Failed

VPC Origins is a relatively recent CloudFront feature that lets customers deliver content from applications running behind private load balancers without exposing origin servers directly to the internet. The architecture is appealing for hardened setups: your back end stays off the public network, and CloudFront handles all public-facing traffic. But that routing path depends on a packet processing subsystem that, on July 16, broke inside availability zone euc1-az2 of the eu-central-1 (Frankfurt) region.

AWS was clear that only customers using VPC Origins were affected. Standard S3 origins, ALB origins, and custom HTTP origins continued serving normally. That's an important distinction — most CloudFront deployments use conventional configurations and saw no disruption. But for the customers who had adopted VPC Origins, a feature AWS actively promotes for security-conscious architectures, the failure was complete. AWS's immediate workaround: if you don't strictly need VPC Origins, switch origin type temporarily while engineers work on a fix.

How One AZ in Germany Affects the Whole World

CloudFront is a global CDN with hundreds of edge locations on every continent. Those edge nodes receive requests from nearby users and forward them to origin servers via shared routing infrastructure. When the VPC Origins routing layer in Frankfurt's euc1-az2 failed, that failure propagated worldwide through CloudFront's request pipeline — edge nodes in Tokyo, São Paulo, and Chicago were all attempting to route through the affected subsystem.

The geographic origin of the problem was one data center in Germany, but its blast radius was global. Among the early confirmed casualties: Hugging Face, the AI developer platform used by machine learning teams worldwide, reported its service was unavailable from most regions. The UK National Lottery confirmed that players couldn't reach its website or mobile app. Japanese streaming services reported widespread 504 errors shortly after the incident started.

Cloud Had a Rough 48 Hours

This was the second significant managed infrastructure failure in two days. On July 14 at 10:00 PDT, Google Cloud VMware Engine (GCVE) suffered more than 12 hours of disruption across four regions — australia-southeast1, australia-southeast2, europe-west3, and northamerica-northeast2 — after a network configuration update caused BGP session failures between availability zones. Virtual machines kept running throughout the incident, but customers couldn't reach them from the internet. GCVE's core value proposition for stretched clusters — high availability across two physical sites — was completely undermined by a single bad configuration push.

Both incidents share the same underlying pattern: a change in a control-plane or routing layer, introduced in a specific region, cascaded further than anyone expected. Neither failure was caused by hardware dying at random; both were brought on by infrastructure changes that propagated poorly through a complex system.

The Uncomfortable Reality of Cloud Reliability

Cloud providers deliver on their SLAs the overwhelming majority of the time, and hyperscale infrastructure is genuinely more reliable than most businesses could build on their own. But reliability most of the time and guaranteed availability are different things. Today's outage illustrates a structural reality that grows more relevant as cloud feature sets expand: the more sophisticated the configuration, the more failure modes it introduces.

VPC Origins is a good example. It solves a real problem — keeping origin servers off the public internet — but it does so by adding a new routing layer with its own operational surface area. Simpler configurations often have better practical resilience precisely because there are fewer moving parts. AWS's own workaround advice — switch origin type if you don't strictly need VPC Origins — points directly at this tradeoff. The security benefit of private origins is real, but so is the dependency on a newer, less battle-tested path.

What Businesses Should Take From This

If you're not a CloudFront VPC Origins customer, today's outage didn't touch you. But the broader lessons apply to any business running a website that depends on CDN or cloud infrastructure:

• Audit your CDN dependency. If a single CDN provider is the only delivery path to your site, you have a single point of failure. Multi-CDN routing is complex and expensive for most small businesses, but DNS-level failover to a secondary provider — or even a static fallback page — for critical paths is achievable and worth planning.
• Test your failover before it's urgent. Having a documented failover procedure means nothing if no one has practiced it. Scheduled drills — even informal ones — surface gaps before production does.
• Monitor upstream infrastructure, not just your own servers. Most basic monitoring setups alert when your origin server is down but miss cases where an upstream CDN, DNS provider, or cloud routing layer is the problem. Synthetic monitoring from external probes in multiple geographic regions catches these scenarios, and is often the difference between detecting an outage in two minutes versus two hours.
• Keep critical delivery paths simple. New CDN features are genuinely useful. But the primary delivery path for your homepage isn't the place to beta-test sophisticated origin routing configurations. Run experimental architectures on less critical endpoints where a failure has bounded impact.

At Falcon Internet, our 24x7x365 NOC monitoring includes upstream infrastructure health checks — not just origin server availability — because as this week demonstrated, a server can be completely healthy while the network layer between it and your customers is broken. A green dashboard on your host's end is cold comfort when your customers are staring at a 504.