nerdylorrin.net » Outages

2010-08-01 Web, mail outage

Lorrin — Mon, 02 Aug 2010 04:37:07 +0000

домейнKiwi ran completely out of memory and crashed hard just before noon (PDT) today. Back up around 9:15 pm (PDT).

Gallery, wiki unaffected (they run on different servers).

Update 2010-08-06:

I’ve belatedly taken some stops to reduce memory usage. Spamassassin and Apache will now fork fewer child processes, which will hopefully keep things under control.

2010-06-14 Gallery Outage

Lorrin — Tue, 15 Jun 2010 06:22:36 +0000

It looks to me like Dreamhost did an upgrade that broke my PHP installation. Gallery was returning HTTP 500 errors from 9 am to 11 pm PDT. Fixed.

2010-05-08 Kiwi upgraded

Lorrin — Fri, 21 May 2010 05:06:05 +0000

(belated notification)

Two weekends ago I upgraded kiwi to the latest Ubuntu release. I think the only visible effect is a major update to the WordPress administrative interface. For those of you using webmail (oh wait, that’s no one outside of this house) the upgrade paves the way major updates there as well. Blogs were down for a couple hours, everything else for a few minutes.

2010-02-26 Web, Mail outage

Lorrin — Sat, 27 Feb 2010 02:52:19 +0000

Kiwi ran out of disk space and memory (perhaps running out of disk space resulted in no room for the swap file to grow?) around 1 am PST this morning. Upgraded both and brought back online around 6:45pm PST.

Web sites (except Gallery and Wiki) and mail were down. Everything should be back to normal now.

2009-10-09 Web outage

Lorrin — Sat, 10 Oct 2009 06:04:41 +0000

Module misconfiguration on kiwi brought Apache down from 9:15 – 11:00 PM (PST). Web sites (except Gallery and Wiki) were down. Everything should be back to normal now.

2009-08-21 Web outage

Lorrin — Sat, 22 Aug 2009 08:04:28 +0000

An upgrade on kiwi didn’t go as smoothly as one might hope. Web sites (except Gallery and Wiki) were down starting around 9 pm PST. Blogs in particular were down for about three hours. Everything should be back to normal now.

2008-06-11 Gallery, Wiki outage

Lorrin — Fri, 12 Jun 2009 06:49:19 +0000

A brief power outage exceeded the UPS’ capacity. Time to get new batteries! Services were down approximately 11:40 p.m. to midnight PDT.

2009-03-27 Web, mail outages

Lorrin — Sat, 28 Mar 2009 06:16:03 +0000

Websites (other than wiki and gallery) and mail were out a few hours this evening (approx 4:00 – 7:15 p.m. PDT ). They’re hosted on kiwi, which locked up and needed to be restarted. Unlike the December outage, this time it was just kiwi and not the parent Xen host. Root cause unclear. Spam Assassin was churning through a bunch of spam at the time, but that could be a random correlation.

2008-12-24 Web, mail outages

Lorrin — Thu, 25 Dec 2008 06:09:19 +0000

Websites (other than wiki and gallery) and mail were out a couple times this afternoon. The parent Xen host crashed twice and was rebooted by RimuHosting. Sigh. Hopefully it’ll stay up this time.

Update 12-25: I must have jinxed it. Host continued to have issues and was out overnight. (Sleigh riding with Santa?) Seems to be back up now after a Xen upgrade.

2008-12-22 Gallery, Wiki outage

Lorrin — Mon, 22 Dec 2008 22:26:12 +0000

Another four months, another outage. Comcast was down for a couple hours, presumably due to snow. Gallery and wiki, which are hosted in the basement on carrot and tomato, were inaccessible.

There was also a gallery and wiki outage about two weeks ago due to an unexpected IP address change.

As usual, mail, blogs, and other websites were unaffected. They’re hosted on kiwi, which is located in a data center in Dallas.