You might have seen some mail trickle in over the last few hours. We've been testing and slowly letting some get through while watching the logs and I just "flipped the switch" to make it go full throttle. Yay!
The nntp service isn't up yet, but will be soon.
Robert got all the data out of onion earlier today, yay.
I have another box sending out the old mail queue from onion now and I'm working on setting up the lists again on one of the boxes we got from BizRate some time ago.
Unless I hit some evil show stopper mail should be flowing again some time later tonight.
Oh, if anyone have some spare 18 or 36GB scsi drives then we could use a few to get mirrored drives. :-)
Yay, progress!
The box couldn't even boot on the 5.2.1 boot CD I made, but it did run on the Linux Rescue CD.
Robert copied the raw partitions with dd to another box so we can mount them there. It's copying the /home partition now and should be done mid-afternoon. Haven't yet decided on a plan for what to do after that. I think we'll boot the box in FreeBSD (4.10 system) and see if it can run stable for a couple of hours while we work on copying (from the raw disk image copies) and setting up the lists on another box.
We are trying to figure out what's making the hardware so flaky. We need it to be able to stay up for at least an hour a few times so we can copy everything over to another box. The fsck takes forever all by itself.
Current theory is that it's something causing a short once in a while, as it's sometimes rebooting shortly after plugging keyboard, monitor or serial port in or out. In the past it'd reboot in the same way if we had a keyboard plugged in (but only then, we thought).
The perl.org MX is actually on another system, so it's (almost) only the mailing lists and the NNTP service that is affected. Mail to RT etc is generally still working.
I made a bunch of boot CDs. I booted it on memtest a couple of hours ago. Thursday morning (PST, thursday afternoon UTC) we'll go down[1] to the data center and see how the memtest went, if it stayed up through it and try to get it up and running one way or another.
When we've succeeded with copying the files I'll set the lists up on our new list server and the mail should start flowing again. Hopefully that'll be sometime late Thursday night or early Friday morning UTC.
[1] It's two floors underground, so literally down...
Steve Fink just reminded me that the tinderbox also used to run on onion. We'll try to get that moved over and setup again somewhere.
Thanks for your patience everyone. :-)
