[specref] Downtime post-mortem from Tobie Langel on 2016-03-04 (spec-prod@w3.org from January to March 2016)

From: Tobie Langel <tobie@codespeaks.com>
Date: Fri, 04 Mar 2016 12:59:32 +0100
To: spec-prod@w3.org
Cc: dom@w3.org
Message-Id: <1457092772.1418772.539492226.52F55FBA@webmail.messagingengine.com>

Hi all,

Sorry Specref was down for a number of hours overnight as I deployed a
buggy app refactor and failed to see that the app was crashing despite
checking the logs.

Lessons learned:
- the app needs tests, not only the data,
- Papertrail logs don't differentiate enough failed and served requests,
so check the app too,
- Don't push code late a night.

Apologies,

--tobie

Received on Friday, 4 March 2016 11:59:57 UTC