Touring - Crazyguyonabike server crash

Hi all,

There seems to have been some kind of hardware failure on the crazyguyonabike server. I got up this morning to find a read-only filesystem which pretty much stops me from doing anything. There are some clues in the logs which would seem to point to something in the RAID or disk controller, or perhaps one or more hard drives. I have put a message in to our hosting company to ask when they can look at this. Since it's the weekend (of course) it's possible that it might be Monday before anything really gets done, but I'll post updates here if I hear anything.

Neil

Thanks Neil for the update. Hope it gets sorted out sooner rather than later. I just finished an 8-day tour here in Alaska with my son, and was looking to get caught up with a few tour logs I'm following.

Ted

Will all the journals, forum posts etc still be there when the problem is worked out?

Ken Payton put so many miles on it it crashed

Will all the journals, forum posts etc still be there when the problem is worked out?

I use MySQL database replication to instantaneously copy updates from the main server in Vegas to two remote computers (one server hosted in Canada, and my home workstation in California). Also, pics (which aren't stored in the MySQL database) are mirrored via a separate cronjob every minute. All data is also separately backed up to an external hard drive here at home. There are also other external hard drives which are backed up to weekly and stored in a firesafe. So even if a complete hardware failure should completely wipe out the server in Vegas, we should have multiple versions of the data in distributed geographical locations. Usually we should be looking at zero data loss, or at worst a minute's worth of pic updates.

So while the data might not still be there on the server's hard drives, it is restorable from elsewhere, probably up-to-the-minute or better.

This doesn't say how long it'll take to bring the server back up, of course - but the data itself should be ok.

Neil

Ken Payton put so many miles on it it crashed

See my reply above - "crashed" does not mean "lost". Hardware crashes; you can't do anything about that, and I currently cannot afford to run multiple hot spares. However you can take steps to make sure the data is safe in the event of a hardware failure, and this I believe I have done. So Ken's journal (and everyone else's) will be just fine.

Of course, I apologise to all for the downtime.

Neil

i think that was a joke^^^^^^^^^^^^^

The server is back up again. It appears to have been some kind of hiccup with the RAID controller losing the connection with one of the drives, rather than an actual hardware failure. I have posted a thread with more of a possible explanation (from the guy who built the server and runs the hosting company where it is currently colocated) on crazyguyonabike.

http://www.crazyguyonabike.com/forum/board/message/?thread_id=124235

Neil

some kind of hiccup with the RAID controller losing the connection with one of the drives

Thought that was the purpose of RAID..so if a drive died it kept running.. But I only do it for a living so what do I know :P

well, that would depend upon the raid.

i would question the controller or interface (backplane, cable, etc.).

Thought that was the purpose of RAID..so if a drive died it kept running.. But I only do it for a living so what do I know :P

See my post over on the crazyguyonabike thread for a possible explanation from Rob Tyree.

i noticed it went down, it happend while i was reading one of the journals "Around the united states on a budget".

Best of luck and thanks for the great site!

well, that would depend upon the raid.

i would question the controller or interface (backplane, cable, etc.).

I guess I'm too used to top end stuff where you can snatch a drive out on the fly :P

But we only see 20mil so a day users, so they're not stressed much :)