Power Outage to Cost RackSpace up to $3.5 million in Refunds

According to a SEC filing by the publicly traded company RackSpace, the power outage that caused its servers to go offline for an extended period last month will cost the company up to $3.5 million USD in refunds. According to the report: “We have experienced power interruptions which have affected a portion of our Grapevine, Texas data center. We have posted updates on our recent power interruption on our website blog and our customer portal for the benefit of our customers. We are continuing to assess the financial impact of service credits due to these events. Currently, our preliminary range for the resulting one time service credits is estimated to be between $2.5 million and $3.5 million. Our website blog is located at http://www.rackspace.com/blog/ .”

The story of RackSpace’s downtime had a nice run across Twitter & the blogosphere, typical of which is this post by TechCrunch: “Last week, Michael Jackson’s death caused sites to fail left and right. Today, it’s a very different problem. The hosting service Rackspace has been completely down for the past 30 minutes or so… Apparently, it’s an entire network outage [Update below, while it was a massive outage, it wasn't a full outage, apparently.] and so the usually very responsive Rackspace team cannot even respond to emails or tweet (though I’m sure we’ll be seeing some updates from smartphones shortly).”

I have been personally considering migrating to Mosso, looking at their cloud sites and cloud servers options, as we also had trouble for sometime on our own host. Bottom line is basically every and any host is going to have issues from time to time, when everything is working normally you rarely stop to appreciate how good the service actually is. But most people have zero tolerance for downtime from a web host, as that is the basic fundamental service they are providing as a business. If the downtime issues continue, mass migration quickly ensues. Did the issue hit a crisis point for RackSpace?

From June 30th, 2009 RackSpace blog:

“Rackspace community,

Yesterday afternoon at 3:15CDT our data center in Dallas experienced an interruption in power to portions of the facility. The interruption caused customer servers to lose power and go down. We sincerely apologize for this disruption and know that it impacted our customers’ businesses as well as the experience of many who use the web. Although we have had some issues with this data center before, please know that we will do what it takes to improve its reliability and performance. We owe you an action plan to prevent this type of thing in the future, and we’ll get that to you as soon as it is ready.

Specific to this situation, here’s what we are doing right now:

The data center is currently running on utility power.

We are continuing to research the root cause analysis for yesterday’s generator failures. We have flown in our senior-level engineers from our global operations, and they are working with our external suppliers to determine the cause and how we can prevent this from happening again. We have the best outside experts from companies like Cummins, GE and Eaton.

We have re-serviced and re-checked our UPS units.

Tonight at 9:00CDT we will continue our testing of the generator bank in question as we narrow down the variables to determine and remediate root cause.

Our Support teams will continue to work with all affected customers to ensure they’re up and running.

We will continue to provide status updates on our customer portal (https://my.rackspace.com/) and on http://www.rackspace.com/blog/.

A copy of the incident report that we sent to affected customers can be found at the following link. Though we typically treat our incident reports as proprietary information between us and our customers, we are publicly posting the report for this incident due to high level of public interest that this incident has received.

I want to ensure you that we are doing everything we can to bring this to resolution as quickly as possible. We appreciate your support and understanding. Our promise is Fanatical Support, we believe in it, and we will work with each of our customers to honor that promise.

Lanham Napier CEO, Rackspace Hosting”

Have they solved the issue?

RackSpace Blog – July 7, 2009

Dallas data center update as of 1:30 pm CDT

“Today at approximately 11:00 AM, an electrical connection failed, causing a brief power interruption to customers on UPS cluster A. This failure also may have caused intermittent network performance issues for customers supported by UPS clusters B and E for a short time. For cluster A customers, we bypassed the UPS and restored power to the servers via generator within a few minutes. Currently systems supported by UPS cluster A are still running on generator power. Repairs are underway and we plan to return to utility power with UPS support as soon as possible. We will follow up with additional updates as new information becomes available.”

RackSpace will undoubtedly lose some business because of this, but they have been having massive growth even during the recession, and it seems they are planning to calm down quite a few customers with a refund or account credit. Since the failures seem to be related to mundane issues related to power generation and backup power supplies, and not the cloud server architecture, I would expect the problem to be easily resolved and hopefully no longer an issue in the future.

Mullenweg: Biggest WordPress.com Mistake – “Buying Servers”

WordPress Guru & Automattic founder Matt Mullenweg gave a speech at the recent GigaOm’s Structure 09 conference in San Francisco, where he spoke on cloud computing and wordpress.com’s server infrastructure. The speech was reported on by Rich Miller at datacenterknowledge.com. According to the report, Mullenweg said, “The biggest mistake we made with the Wordpress.com infrastructure was actually buying servers… (buying servers was ) not a utility. Now we lease them all on a month-to-month basis.” According to the presentation, WordPress.com runs “about 5 million sites serving more than 1 billion page views a month. Automattic uses two data center providers, the dedicated hosting specialists ServerBeach (PEER 1) and Layered Technologies.”

Mullenweg went on to say he viewed WordPress.com’s use of Amazon’s S3 storage system “a failure” because it represented a lack of an open source alternative. “When I have to go to the cloud, I consider that a failure. The thing that’s been most exciting to me is how the open source tools have evolved.” The summary, for those who would like to learn from the experience of running one of the most highly trafficked, multi-user blog sites on the internet? Use leased servers rather than investing in your own data center, that way you can keep on the latest and most up to date boxes without having to cover the cost of the perpetual upgrades, management staff for the data center, and infrastructure / backup facilities.

Month to month leasing also keeps open the negotiation options for a large hosting account like WordPress when dealing with service providers, rather than getting locked into a long term contract or HR staffing issue. This makes a lot of sense. For example, I recently looked into opening a data center in India based around Virident Eco Ram servers, 4 at around $5000 each. Add office space in an Indian IT Park ($400-$500 p/month), backup power supply and power synchronizers ($5000), cooling system ($2500), 24 hour sys admin staff ($36-$40,000 p/year), internet backbone connection ($1250-$2000 p/year?), utilities ($100-$150 p/month), etc. and the total quickly adds up. Granted you likely do not need 24 hour sys admin for 4 servers, but how to maintain a data center without on-site security and maintenance? Compare this total cost to what you would receive over a similar time period with leased servers. Easy math.

This is why Mullenweg ended up advising start-ups, small development companies, and media groups not to try to compete with what the major tech companies are doing with web infrastructure. “My challenge to everyone competing with Amazon, Google and Microsoft is to remember that you’re competing with Amazon, Google and Microsoft,” he said. “These are strong technology companies, and if you’re going to compete with them, open source is the only way to do that. Otherwise, you have no leverage.”

What then is the “open source equivalent” of a cloud server, when one of the main aspects to the cloud is “utility banks” of grid servers that can expand and contract around the traffic needs of an individual site? With shared, leased, or dedicated servers you are renting a fixed amount of disk space or a set number of machines. It is the ability to scale to meet the highest peak demand, digg-effect, etc. that the cloud is delivering through mass, corporate data banks. Unless someone develops a “distributed computing” model of sharing cpu resources across a network to freely scale in times of peak traffic, and many “open source” servers joined together to share resources in this way, I don’t see an exact “open source equivalent” to the grid. Maybe it is Bittorrent, LimeWire P2P networks, but the performance there is a lot different from Akamai or RackSpace.

Mullenweg recommended the nginx web server for load balancing:

“nginx has been running for more than four years on many heavily loaded Russian sites including Rambler (RamblerMedia.com).
In March 2007 about 20% of all Russian virtual hosts were served or proxied by nginx.
According to Google Online Security Blog year ago nginx served or proxied about 4% of all Internet virtual hosts.
2 of Alexa US Top100 sites use nginx.
According to Netcraft in December 2008 nginx served or proxied 3.5 millions virtual hosts. And now it is on 3rd place (not counting in-house Google server) and ahead of lighttpd.
According to Netcraft in March 2009 nginx served or proxied 3.06% busiest sites.
According to Netcraft in May 2009 nginx served or proxied 3.25% busiest sites.
Here are some of success stories: FastMail.FM, Wordpress.com.”

Source: http://nginx.net/

Optimizing a dedicated server for a high traffic site and “cloud hosting” are very different undertakings. An open source alternative to this aspect of “the cloud” would involve users sharing their CPU cycles with other users around the world or locally on the same network during peak traffic or spinning off virtual clones of the site during overflow to another machine to handle the load. There would seem to be a number of security issues that would arise, and ultimately, somewhere there has to be charity – people giving up their processing power and bandwidth when it is not being used, sharing empty, allocated disk space – entire websites at the end of a torrent, mirrored on different servers. Right now it is basically the SETI Screensaver as a model of the open cloud.

If you have your own data center or dedicated cluster, Apache Hadoop is an open source distributed option: http://hadoop.apache.org/core/

Related: http://www.morganclaypool.com/doi/abs/10.2200/S00193ED1V01Y200905CAC006

Read the full Article: http://www.datacenterknowledge.com/archives/2009/06/25/mullenweg-open-source-trumps-the-cloud/

DowntownHost – Recommended Shared, Linux Web Hosting

Over the years, I have used Yahoo, Hostway, Sharkspace, and Godaddy, sequentially, for my web hosting services until finally changing to Downtownhost in 2008. Yahoo was great during the Web 1.0 years, but became expensive in comparison to other services and I migrated to Hostway around 2001, using its servers to run an online art gallery for over 6 years. When I launched a new web development company in 2006, I shifted to SharkSpace hoping to use their reseller hosting package as a good way to manage disk space for multiple clients, but the price and disk space could not compare to shared hosts like Godaddy. (For reference, $24.95 per month vs. $5 per month.) I signed up for a Wild West reseller account thinking that would be a better option for hosting client sites cheaply – big mistake. I had innumerable problems with Godaddy doing CMS development and hosting multiple domains on the same shared space. So many problems in fact, I decided I could not offer the services responsibly to my clients or even use the disk space for my own sites.

I canceled my Godaddy account and began a serious search of all hosts for the best one to build my own sites on as well as host my client’s sites reliably, securely, and economically. I checked all of the main web hosting companies, all of the review sites, forums, bulletin boards, etc. A lot of people had good things to say about DowntownHost, something that is really rare if you look at discussions of hosting companies. One thing that was mentioned, is that when hosting companies offer disk space in massive quantities (500GB, Unlimited Storage) and unlimited bandwidth for a super cheap price, they are overselling their servers – i.e. selling disk space and bandwidth they don’t actually have, and betting that most users won’t even use a fraction of their 500GB. That is fine, it probably wouldn’t happen anyway, but then the same shared hosting companies have the server processor use restrictions and can shut your site down even if they promise unlimited bandwidth. In a nutshell, you get what you pay for, and with cheap shared hosting a lot of the servers can develop small problems with configurations that cause nightmares for developers. Godaddy is the worst example here probably, just check any CMS board about the host-specific problems developers discover there.

Read the rest of DowntownHost – Recommended Shared, Linux Web Hosting »

Cloud hosting with the Rackspace Cloud

You may have seen the announcement on twitter (if not then shame on you, you should be following us). WebDevNews has moved to new hosting. As Xavisys Web Development (the company that owns/operates this site) has been growing, we’ve been trying to find just the right company to partner with in order to offer high quality hosting to our clients. Rackspace Cloud Hosting is that company. We tried quite a few ranging from Bluehost to Host Gator to Rackspace. We were not looking for cheap hosting, we were looking for the best hosting we could find for our specific needs. While some were better than others, RackSpace Cloud Hosting was the best.

In order to test out the hosting, we moved a couple sites to the Rackspace Cloud. We didn’t want to move any really big complex sites, but didn’t exactly want to test with my wife’s blog either. Instead, we moved both WebDevNews and Attackr. Both are WordPress (database driven) sites, and combined they average about 650 pageviews a day. While it’s not a lot of traffic, we thought it would give us a good idea of how Rackspace Cloud Hosting would perform.

Read the rest of Cloud hosting with the Rackspace Cloud »