Saturday, August 16, 2008

3 days of not selling burgers

Wow, a 3 day outage at Netflix. I know this is a blog about performance (specifically around WebSphere Application Server) but I have a fascination with production outages because I normally work those kinds of problems. They always revolve around human error either by fumble fingering something or not executing (like not conducting performance testing).

But a three day outage is excessive. That must have been a real interesting problem because I've never seen an outage that long (at least not after I have arrived to fix it). And it'll cost Netflix an estimated $6 million.

As the article states, imagine if McDonald's couldn't see burgers for 3 days. BTW, the McD's in Dwight, IL off I-55 lost their cash registers one day a couple of weekends ago when I stopped in to get some food on my long drive to Chicago. It was interesting to see how dependent McD's is on that register system because it drives their whole operations and what the human equivalent (i.e. yelling orders into the back area) and having to use calculators and paper/pen to record how many of each item was sold. Needless to say the credit card readers were down too so if you didn't have cash you were going hungry that day.

Lessons From Netflixs Fail Week - Bits - Technology - New York Times Blog
Netflix, the DVD-by-mail service, largely ceased shipping DVDs to its 8.4 million subscribers for three days this week. The company vaguely blames a technology glitch.

No comments: