Tag Archives: sysadmin

Tweet Marker SSL mistake

It usually takes a couple problems hitting at once to cause a major server outage. This happened last week when Tweet Marker’s SSL certificate expired. I have the SSL set to auto-renew, but it still requires manually installing the new certificate, and other problems happened along the way.

First mistake: I didn’t realize it was expiring. Those emails go to an account I don’t check very often, littered with spam. And the email to confirm the renewal went to yet another email address that no longer worked. When I had moved the DNS hosting to Amazon’s Route 53, I had neglected to move over the MX records.

After fixing all of that, I tried updating the app on Heroku to use the new cert, only to get stalled as Heroku’s new SSL add-on rejected it. Certain I had done something wrong, I fumbled through a dozen Heroku SSL how-to posts before finally reverting to their old SSL add-on. It’s no longer documented and is in fact actively discouraged by Heroku, but it also has the lucky trait of actually working with my certificate. Updating DNS caused another hour-long delay because of the high TTL.

I sent two support requests during this process, so I thought I’d rate how each company did:

  • DreamHost: Before I figured out the bad email address, I sent DreamHost a question about why the SSL certificate hadn’t showed up yet. They responded very quickly, and even included a “P.S.” that they were fans of Tweet Marker. Basically they provided excellent support, the best you could ask for.

  • Heroku: When the new SSL add-on wasn’t accepting my certificate, I filed a support request with Heroku as well. The response was an automated reply that they don’t do support past 6pm. For a hosting company that charges a premium, this was a disappointing response. (They responded first thing the next morning, though.)

This SSL glitch was the only significant outage Tweet Marker has had in its first year. I learned a few lessons, took the opportunity to check backups and EC2 servers, and now I’m ready to move on. Hoping for an even better year 2.

Dreamhost scale

I get a lot of funny looks when I tell people I host everything on Dreamhost. It’s not a great fit for everything — I have some ideas for projects that would be better suited to Amazon EC2, and who knows, maybe I’ve just been on a lucky server — but it has generally been more reliable than any previous hosting company I’ve used, including when I used to run my own server.

Dreamhost succeeds because of scale. They have so many servers, and such low prices, that they are forced to automate everything. This means they can more quickly deploy new software, rebuild servers, or restore a broken installation, and that their panel interface has to provide access to every feature a customer might want.

“This post from their status blog”:http://www.dreamhoststatus.com/2009/07/18/network-problems-due-to-distribution-switch/ is revealing. There are over 600 machines on that list, but it must be only some fraction of their customer base, because my server name isn’t on there. “According to WebHosting.Info”:http://www.webhosting.info/webhosts/reports/total_domains/DREAMHOST.COM, Dreamhost hosts about 875,000 domains.

I strongly believe that “being small is a competitive advantage”:http://www.manton.org/2007/02/customer.html, but anyone who’s played the role of sys admin knows that automation means everything, and that’s what Dreamhost seems to get right.