Temporarily Unavailable

LiveJournal: LiveJournal is currently down due to a massive power failure at our data center. We'll provide updates at /powerloss/ as they're available.

• Our data center (Internap, the same one we've been at for many years) lost all its power, including redundant backup power, for some unknown reason.
• Update #1, 7:35 pm PST: we have power again, and we're working to assess the state of the databases.
• Update #2, 10:11 pm: So far so good.
• Update #3: 2:42 am: We're starting to get tired, but all the hard stuff is done at least.
• Update #4: 9:12 am: We're back at it.
• Update #5: 1:58 pm: approaching 24 hours of downtime... *sigh*

Randy: Yikes! Can't wait to learn from the post-mortem. Reliability of scalable systems.