« Hadoop versus VPN | Main | Glue Fleet and Compojure Together Using Protocols »

GMail Outage Was a Chain Reaction

Posted by: Michael Nygard on 09/02/2009

Google has published an explanation of the widespread GMail outage from September 1st. In this explanation, they trace the root cause to a layer of "request routers":

...a few of the request routers became overloaded and in effect told the rest of the system "stop sending us traffic, we're too slow!". This transferred the load onto the remaining request routers, causing a few more of them to also become overloaded, and within minutes nearly all of the request routers were overloaded.

This perfectly describes the "Chain Reaction" stability antipattern from Release It!

Michael Nygard's complete blog can be found at: http://www.michaelnygard.com/blog/index.html

About Michael Nygard

Michael strives to raise the bar and ease the pain for developers across the country. He shares his passion and energy for improvement with everyone he meets, sometimes even with their permission. Michael has spent the better part of 20 years learning what it means to be a professional programmer who cares about art, quality, and craft. He's always ready to spend time with other developers who are fully engaged and devoted to their work--the "wide awake" developers. On the flip side, he cannot abide apathy or wasted potential.

Michael has been a professional programmer and architect for nearly 20 years. During that time, he has delivered running systems to the U. S. Government, the military, banking, finance, agriculture, and retail industries. More often than not, Michael has lived with the systems he built. This experience with the real world of operations changed his views about software architecture and development forever.

He worked through the birth and infancy of a Tier 1 retail site and has often served as "roving troubleshooter" for other online businesses. These experiences give him a unique perspective on building software for high performance and high reliability in the face of an actively hostile environment.

Most recently, Michael wrote "Release It! Design and Deploy Production-Ready Software", a book that realizes many of his thoughts about building software that does more than just pass QA, it survives the real world. Michael previously wrote numerous articles and editorials, spoke at Comdex, and co-authored one of the early Java books.

More About Michael »