Unplanned Site Downtime

Hi, folks–

Around 21:50 ET last night (Sunday), the site went down (all services 
included). The downtime lasted for almost 30 minutes.

Renoir responded promptly, but was unable to connect to the server, so 
he couldn't do anything. The site came back up on its own.

Renoir contacted Dreamhost, and they informed us that there was a DDOS 
attack on the entire data center [1], which was the cause for our 
downtime. This was not related to our site's stability, per se, which is 
the good news; our local infrastructure was not affected; this was just 
a network issue.

In order to try to prevent problems like this in the future, Renoir and 
I have a two-stage plan for future stability:

1) Short-term: Attempt to configure our CDN, Fastly, to continue to 
serve static pages; this will not let us edit pages, but it will let 
readers continue to use the site;

2) Longer-term: Work with an additional web host to distribute our site 
across two different services (which would of course be on two different 
data centers); theoretically, this is one of the strengths of OpenStack.

This was not a particularly serious outage, coming at the time it did 
and not lasting too long, but we do want to work to prevent issues like 
this in the future.

We apologized for any inconvenience this may have caused you.

[1] 
http://www.dreamhoststatus.com/2014/02/16/connectivity-issues-with-us-east-virginia-data-center-2/

Regards-
-Doug

Received on Monday, 17 February 2014 19:31:09 UTC