The story of RackSpace’s downtime had a nice run across Twitter & the blogosphere, typical of which is this post by TechCrunch: “Last week, Michael Jackson’s death caused sites to fail left and right. Today, it’s a very different problem. The hosting service Rackspace has been completely down for the past 30 minutes or so… Apparently, it’s an entire network outage [Update below, while it was a massive outage, it wasn't a full outage, apparently.] and so the usually very responsive Rackspace team cannot even respond to emails or tweet (though I’m sure we’ll be seeing some updates from smartphones shortly).”

I have been personally considering migrating to Mosso, looking at their cloud sites and cloud servers options, as we also had trouble for sometime on our own host. Bottom line is basically every and any host is going to have issues from time to time, when everything is working normally you rarely stop to appreciate how good the service actually is. But most people have zero tolerance for downtime from a web host, as that is the basic fundamental service they are providing as a business. If the downtime issues continue, mass migration quickly ensues. Did the issue hit a crisis point for RackSpace?

From June 30th, 2009 RackSpace blog:

“Rackspace community,

Yesterday afternoon at 3:15CDT our data center in Dallas experienced an interruption in power to portions of the facility. The interruption caused customer servers to lose power and go down. We sincerely apologize for this disruption and know that it impacted our customers’ businesses as well as the experience of many who use the web. Although we have had some issues with this data center before, please know that we will do what it takes to improve its reliability and performance. We owe you an action plan to prevent this type of thing in the future, and we’ll get that to you as soon as it is ready.

Specific to this situation, here’s what we are doing right now:

The data center is currently running on utility power.

We are continuing to research the root cause analysis for yesterday’s generator failures. We have flown in our senior-level engineers from our global operations, and they are working with our external suppliers to determine the cause and how we can prevent this from happening again. We have the best outside experts from companies like Cummins, GE and Eaton.

We have re-serviced and re-checked our UPS units.

Tonight at 9:00CDT we will continue our testing of the generator bank in question as we narrow down the variables to determine and remediate root cause.

Our Support teams will continue to work with all affected customers to ensure they’re up and running.

We will continue to provide status updates on our customer portal (https://my.rackspace.com/) and on http://www.rackspace.com/blog/.

A copy of the incident report that we sent to affected customers can be found at the following link. Though we typically treat our incident reports as proprietary information between us and our customers, we are publicly posting the report for this incident due to high level of public interest that this incident has received.

I want to ensure you that we are doing everything we can to bring this to resolution as quickly as possible. We appreciate your support and understanding. Our promise is Fanatical Support, we believe in it, and we will work with each of our customers to honor that promise.

Lanham Napier CEO, Rackspace Hosting”

Have they solved the issue?

RackSpace Blog – July 7, 2009

Dallas data center update as of 1:30 pm CDT

“Today at approximately 11:00 AM, an electrical connection failed, causing a brief power interruption to customers on UPS cluster A. This failure also may have caused intermittent network performance issues for customers supported by UPS clusters B and E for a short time. For cluster A customers, we bypassed the UPS and restored power to the servers via generator within a few minutes. Currently systems supported by UPS cluster A are still running on generator power. Repairs are underway and we plan to return to utility power with UPS support as soon as possible. We will follow up with additional updates as new information becomes available.”

RackSpace will undoubtedly lose some business because of this, but they have been having massive growth even during the recession, and it seems they are planning to calm down quite a few customers with a refund or account credit. Since the failures seem to be related to mundane issues related to power generation and backup power supplies, and not the cloud server architecture, I would expect the problem to be easily resolved and hopefully no longer an issue in the future.

This is a post from Web Dev News, a site brought to you by Xavisys Web Development.

Power Outage to Cost RackSpace up to $3.5 million in Refunds

Mullenweg went on to say he viewed WordPress.com’s use of Amazon’s S3 storage system “a failure” because it represented a lack of an open source alternative. “When I have to go to the cloud, I consider that a failure. The thing that’s been most exciting to me is how the open source tools have evolved.” The summary, for those who would like to learn from the experience of running one of the most highly trafficked, multi-user blog sites on the internet? Use leased servers rather than investing in your own data center, that way you can keep on the latest and most up to date boxes without having to cover the cost of the perpetual upgrades, management staff for the data center, and infrastructure / backup facilities.

Month to month leasing also keeps open the negotiation options for a large hosting account like WordPress when dealing with service providers, rather than getting locked into a long term contract or HR staffing issue. This makes a lot of sense. For example, I recently looked into opening a data center in India based around Virident Eco Ram servers, 4 at around $5000 each. Add office space in an Indian IT Park ($400-$500 p/month), backup power supply and power synchronizers ($5000), cooling system ($2500), 24 hour sys admin staff ($36-$40,000 p/year), internet backbone connection ($1250-$2000 p/year?), utilities ($100-$150 p/month), etc. and the total quickly adds up. Granted you likely do not need 24 hour sys admin for 4 servers, but how to maintain a data center without on-site security and maintenance? Compare this total cost to what you would receive over a similar time period with leased servers. Easy math.

This is why Mullenweg ended up advising start-ups, small development companies, and media groups not to try to compete with what the major tech companies are doing with web infrastructure. “My challenge to everyone competing with Amazon, Google and Microsoft is to remember that you’re competing with Amazon, Google and Microsoft,” he said. “These are strong technology companies, and if you’re going to compete with them, open source is the only way to do that. Otherwise, you have no leverage.”

What then is the “open source equivalent” of a cloud server, when one of the main aspects to the cloud is “utility banks” of grid servers that can expand and contract around the traffic needs of an individual site? With shared, leased, or dedicated servers you are renting a fixed amount of disk space or a set number of machines. It is the ability to scale to meet the highest peak demand, digg-effect, etc. that the cloud is delivering through mass, corporate data banks. Unless someone develops a “distributed computing” model of sharing cpu resources across a network to freely scale in times of peak traffic, and many “open source” servers joined together to share resources in this way, I don’t see an exact “open source equivalent” to the grid. Maybe it is Bittorrent, LimeWire P2P networks, but the performance there is a lot different from Akamai or RackSpace.

Mullenweg recommended the nginx web server for load balancing:

“nginx has been running for more than four years on many heavily loaded Russian sites including Rambler (RamblerMedia.com).
In March 2007 about 20% of all Russian virtual hosts were served or proxied by nginx.
According to Google Online Security Blog year ago nginx served or proxied about 4% of all Internet virtual hosts.
2 of Alexa US Top100 sites use nginx.
According to Netcraft in December 2008 nginx served or proxied 3.5 millions virtual hosts. And now it is on 3rd place (not counting in-house Google server) and ahead of lighttpd.
According to Netcraft in March 2009 nginx served or proxied 3.06% busiest sites.
According to Netcraft in May 2009 nginx served or proxied 3.25% busiest sites.
Here are some of success stories: FastMail.FM, WordPress.com.”

Source: http://nginx.net/

Optimizing a dedicated server for a high traffic site and “cloud hosting” are very different undertakings. An open source alternative to this aspect of “the cloud” would involve users sharing their CPU cycles with other users around the world or locally on the same network during peak traffic or spinning off virtual clones of the site during overflow to another machine to handle the load. There would seem to be a number of security issues that would arise, and ultimately, somewhere there has to be charity – people giving up their processing power and bandwidth when it is not being used, sharing empty, allocated disk space – entire websites at the end of a torrent, mirrored on different servers. Right now it is basically the SETI Screensaver as a model of the open cloud.

If you have your own data center or dedicated cluster, Apache Hadoop is an open source distributed option: http://hadoop.apache.org/core/

Related: http://www.morganclaypool.com/doi/abs/10.2200/S00193ED1V01Y200905CAC006

Read the full Article: http://www.datacenterknowledge.com/archives/2009/06/25/mullenweg-open-source-trumps-the-cloud/

This is a post from Web Dev News, a site brought to you by Xavisys Web Development.

Mullenweg: Biggest WordPress.com Mistake – “Buying Servers”

I canceled my Godaddy account and began a serious search of all hosts for the best one to build my own sites on as well as host my client’s sites reliably, securely, and economically. I checked all of the main web hosting companies, all of the review sites, forums, bulletin boards, etc. A lot of people had good things to say about DowntownHost, something that is really rare if you look at discussions of hosting companies. One thing that was mentioned, is that when hosting companies offer disk space in massive quantities (500GB, Unlimited Storage) and unlimited bandwidth for a super cheap price, they are overselling their servers – i.e. selling disk space and bandwidth they don’t actually have, and betting that most users won’t even use a fraction of their 500GB. That is fine, it probably wouldn’t happen anyway, but then the same shared hosting companies have the server processor use restrictions and can shut your site down even if they promise unlimited bandwidth. In a nutshell, you get what you pay for, and with cheap shared hosting a lot of the servers can develop small problems with configurations that cause nightmares for developers. Godaddy is the worst example here probably, just check any CMS board about the host-specific problems developers discover there.

After reading so many reviews and comments, I got in touch with DowntownHost and they have a great service where they will migrate your site files from the existing host to their servers for free when you sign up for a new account. So, the transfer of all my domains from Godaddy was really easy, and free. The main point people make about DowntownHost is customer service. If you have ever submitted a support ticket to a web hosting provider about configuration and development issues you know how important this is and what a nightmare – in my opinion, it is not uncommon to wait 24+ hours for a canned email stating “we can’t help with development issues” on a server configuration related issue that is urgent to deadline-based site development, and no real hope of any assistance being the norm. This is definitely not the case with DowntownHost – any time I have had a question, problem, or issue arise on any topic, and sent an email with a question, they generally replied within 2 to 5 minutes, any time of the day. And they helped to definitively diagnose, repair, and fix the problem.

Downtownhost offers shared Linux web hosting at around 5GB disk space / 100 Gb monthly transfer with a Cpanel interface, free AWStats, unlimited MySQL databases, and unlimited domains at around $5.95 per month, paid a year in advance. As stated, the disk space is not as much as similar web hosting providers, but I found myself migrating away from those shared hosts because of development issues. I find the server configuration at Downtownhost to be perfect for working on multiple CMSs and multiple domains at the same time without any problems. Cpanel is much preferred over the proprietary control panels that you find at many of the web hosting providers, it is very solid, and with all the extras. Cpanel is much more quick and efficient than Dreamhost, MediaTemple, or Godaddy’s administrative backend for example.

As I am doing over 90% Drupal development professionally, I needed to make sure the servers were optimized for Drupal – that was my main reason for doing a quick and extremely dissatisfied flight mid-contract from Godaddy. Working with Pligg and Drupal sites, I was having a lot of trouble with Mod Rewrite and Clean Urls. After solving the problem with one site, it would reappear on another, in a different way, even on a clean install – with no support from Godaddy on any issue. For the record, I have never had a problem with Clean URLs or Mod Rewrite at DowntownHost.

Otherwise, Cpanel includes MySQL Wizard & Management, phpMyAdmin, File Manager, multiple ftp and email accounts for each domain, SSH Access, Mime Types, Apache Handlers, Perl, CGI, Ruby, Ruby on Rails, Fantastico, etc. I requested Python installation on the server but that is probably not happening at DowntownHost anytime soon, unless you upgrade to a private server and install it yourself.

DowntownHost has data centers in both Denver, Colorado and Detroit, Michigan, as I understand it. Before signing up for an account, I went to a site providing server speed benchmarking tests like:

http://www.web-hosting-top.com/web-hosting/tools.website-speed-test

Typing in the names of the top 10 web hosting providers I was evaluating for the test:

downtownhost.com
godaddy.com
mediatemple.net
hostmonster.com
hostgator.com
anhosting.com
rackspace.com
mosso.com
dreamhost.com
1and1.co.uk
SPEED TEST – RESULTS:

However you do the math, DowntownHost tests as the fastest of all of the web hosting providers listed above. I performed the same test with multiple web hosting companies 8 months ago when I switched – and the results haven’t changed much. So, with DowntownHost you have the best customer service in the web hosting industry plus also the top speeds – this is why I consider the company a “hidden gem” and recommend them to my clients and friends.

Note: you may doubt the speed tests above, as they are actually downloading the hosting company’s homepage and testing that speed. When I first joined, DowntownHost also sent me some fixed-sized, set MB number files that could be used for speed benchmarking of the different servers themselves. I do web design from India, and the DowntownHost servers are located in the USA – I have not had any speed or performance related issues arise in 8 months of development work on the servers.

While I haven’t used DowntownHost’s dedicated or VPS servers, I would consider them if I need to upgrade because in my opinion it is difficult to find good customer service and friendly people you can contact repeatedly for reliable tech support in the web hosting industry. That said, I am looking for cloud hosting solutions currently to offer my clients who need web sites that scale and survive the Digg effect or traffic spikes. For that, we are currently evaluating between Amazon and Mosso.

Give DowntownHost a try.

This is a post from Web Dev News, a site brought to you by Xavisys Web Development.

DowntownHost – Recommended Shared, Linux Web Hosting

Use caution when moving to Mosso or the Rackspace Cloud. Everything was great at first, but things went downhill quickly. We started having lots of “No suitable nodes are available to serve your request” errors, which relate to their MySQL servers. Please test thoroughly before using for anything important.

***Warning!***

You may have seen the announcement on twitter (if not then shame on you, you should be following us). WebDevNews has moved to new hosting. As Xavisys Web Development (the company that owns/operates this site) has been growing, we’ve been trying to find just the right company to partner with in order to offer high quality hosting to our clients. Rackspace Cloud Hosting is that company. We tried quite a few ranging from Bluehost to Host Gator to Rackspace. We were not looking for cheap hosting, we were looking for the best hosting we could find for our specific needs. While some were better than others, RackSpace Cloud Hosting was the best.

In order to test out the hosting, we moved a couple sites to the Rackspace Cloud. We didn’t want to move any really big complex sites, but didn’t exactly want to test with my wife’s blog either. Instead, we moved both WebDevNews and Attackr. Both are WordPress (database driven) sites, and combined they average about 650 pageviews a day. While it’s not a lot of traffic, we thought it would give us a good idea of how Rackspace Cloud Hosting would perform.

We hit a few snags along the way. The architecture of the Rackspace Cloud caused WP-Super-Cache (a popular WordPress plugin) to fail. Believe it or not, I was pretty happy about that. See, sometimes you go months without any problems on a host and by the time you have a problem you’re already pretty committed to using them (all your sites are there, etc). The truth is, you will eventually have problems, and I wanted to see how they handled it. I contacted them via the live support chat link in my admin section and got a person within seconds (this seems to be pretty average there). A ticket was filed and they went to work on it. I worked with them, testing, allowing them to modify my .htaccess, etc. As it turned out, there were actually a couple problems. The first was a problem with the rewrite rules and the second was a problem with cleaning up the files. Both are now fixed, and the technicians really went above and beyond in order to fix the problem. I even got calls from them with updates!

While the two sites aren’t huge, they aren’t small either. According to the statistics I could host roughly 18 similar sites for the base rate of $100/month before I would need to start paying overages. Considering the network they offer and the quality of their service, that’s an amazing price!

Sounds like it’s all roses right? Well, there are some drawbacks. For one, they don’t offer Java/Tomcat or ColdFusion. I don’t use those, so that didn’t bother me at all (and they do offer Windows or Linux, PHP, ASP, .Net, Perl, Python, etc). However, I use SSH constantly, and they don’t offer that. I’m adjusting. The best things about SSH is that I could use wget to pull files directly to the server and then uncompress them quickly. For now I’m settling for uploading the file via FTP and uncompressing with their admin panel. Now if they could just find a way to let me use wget!

So in the end, why did we choose them? First and foremost is their service. Of all the companies we tried, Rackspace and Rackspace Cloud (previously called Mosso) had the best service by far. With Rackspace a human answers the phone and immediately puts you through to a knowledgeable technician. With the Rackspace Cloud, you get a recording and have to press 1 to get a technician. While I’d rather have a human answer the phone, at least you can get through to a knowledgeable person in seconds.

Second, they have great reliability. I’ve worked with Rackspace for a while now, and I know their data-centers are extremely reliable. Since their cloud offerings use those same data-centers, they get the same great reliability and connectivity.

Additionally, they scale very easily. With Rackspace, you need to scale in chunks, adding a new server each time. With the Rackspace Cloud we can scale smoothly, which is more cost-effective.

Lastly, they offer some nice little extras. For a little extra per customer per month, they will extend their 24×7 support to my clients. We don’t want to be a hosting company, we just want to be able offer reliable hosting to our clients that need it. Additionally, if you aren’t set up to take recurring payments (or maybe you don’t even accept credit card?!?), they will bill your clients and deposit the money into your account.

If you’re looking for high-quality hosting, either for your sites or to offer to your clients, take a look at Rackspace Cloud Hosting. If you enter the code REF-XAVISYS when you sign up, you’ll get a $25 rebate off your first month!

This is a post from Web Dev News, a site brought to you by Xavisys Web Development.

Cloud hosting with the Rackspace Cloud