Here's a very, very interesting survey that Emerson Network Power commissions every year about data center outages. It doesn't focus on Service Providers so I didn't want to inject it into the previous post about AWS, but some of the answers are stunning (to me). I used to work at one of the companies that was invited to participate every year in this survey, so I've been keeping an eye on these since 2008 or so...
68% of the respondents disagreed with the statement: "We utilize all best practices in data center design and redundancy to maximize availability." Yet 65% say their business model is dependent upon the data center to generate revenue! This is a survey of 453 individuals who have data center operations responsibility at U.S. companies, and less than half (42%) think management supports their efforts to prevent and manage unplanned outages.
The stats on row and rack based outages are incredible. Larger scale, there were an average of 2.48 complete data center shutdowns per respondent over a two year period, with each outage averaging almost two hours (107 minutes), and these are at data centers run by some of the best people in the business.
This is not an easy game to play, even with the basic building blocks of the data center.