Tech

Fast.ly broke the Internet for an hour this morning

The United Kingdom government's official website was one of those affected by this morning's outage. The cryptic "Guru Mediation" message shown is an untrapped, unskinned error returned from the Varnish cache server powering the Fastly CDN.

Enlarge / The United Kingdom government’s official website was one of those affected by this morning’s outage. The cryptic “Guru Mediation” message shown is an untrapped, unskinned error returned from the Varnish cache server powering the Fastly CDN. (credit: Leon Neal via Getty Images)

For roughly an hour this morning—6 am to 7 am EDT, give or take a few minutes—enormous swathes of the Internet were down or interestingly broken. Sites taken down included CNN, The Guardian, The New York Times, PayPal, and Spotify, among many more—including The Verge, which resorted to reporting via Google Docs during the duration of the outage.

The underlying problem was an outage at Fastly, one of the world’s largest Content Delivery Network providers—the entire service went down due to a misconfiguration which it had deployed to all of its Points Of Presence (POPs) globally. As a result, sites using Fastly for content delivery came up with various errors dependent on the local site configuration. Some sites delivered relatively uninformative, plain HTTP 503 (Service Unavailable) pages, while others returned errors such as “Fastly error: unknown domain.”

The “unknown domain” error gives us some tantalizing hints to the nature of the problem, which is more than Fastly’s own status updates have so far. This tells us that Fastly’s network was up and its Varnish cache servers were answering requests, but its cache configuration—the Varnish Control Language files which point the cache server to the back-end servers supplying the original content—was almost certainly either missing or garbled.

Read 3 remaining paragraphs | Comments