neilfein

The site's been down all day. I need my fix!

jamawani 

It is off line for a rebuild.
Will be up tomorrow afternoon.

Canadiense

Here is the official notice from Neil

Website going down for server rebuild on July 16th

Short version for non-geeks:

The website will go down sometime before midday on Wednesday July 16th 2008, for system maintenance. I'll make sure I have everything backed up, then I'll wipe the drives and rebuild the system and data. During the downtime, everything will be unavailable, including crazyguyonabike, the topicwise sites, and the email server. The website will probably be back up again sometime on Thursday. Sorry for the hassle!

Slightly longer explanation for geeks:

The purpose here is to reconfigure the hard drives from RAID10 to RAID0 in order to give us twice as much space as we have now, and faster disk IO as well. To compensate for the lower reliability of RAID0, I'll also be installing a new external 320GB USB hard drive which will be used as a local backup (I can't use an internal drive, since there's no spare space in the 1U server case). In the process, I'll also be installing the newer version of the Debian operating system, namely Lenny (I will be testing this at home prior to the trip). Lenny has newer software than Etch, the version we're currently using, and will soon become the "stable" version of Debian anyway.

I'll be on the train back to St Louis from 7pm up until midnight on the 16th; so if the site is not back up again by late afternoon, then it probably won't happen until the next day. I'll try to post any updates on unexpected delays on the Touring forum over at www.bikeforums.net.

Hopefully people on the road or otherwise updating journals currently will see this and not be too mystified about the downtime. No data should be lost - we have multiple backups on different hard drives.

I wanted to do this while I still have relatively easy access to the datacenter in Chicago. Once we move from St Louis over to Oregon this August, quick trips become a bit harder to do. I may eventually see about moving the server to a datacenter somewhere in California or Oregon, but I'm going to leave that until we are actually over there. We've had good service from our current datacenter, it's fairly cheap and Chicago is not prone to natural disasters such as earthquakes, tornadoes, hurricanes, wildfires, floods, mudslides, volcanoes, tsunamis... it's actually a pretty good place to have a server!

Anyway, thanks for your patience, let's hope it all goes smoothly...

Neil

eastbaybob

I'm with you brother. Things like this make me realize why crazyguy is my one non-tax deductible contribution each year.

chicharron

I'll be glad when its back up again. Ross and Justin are somewhere in Idaho, the last time they posted two days ago. They started in Kitty Hawk NC, some weeks ago, and I have been following them. I pray for their safe arrival in San Francisco.

Neil_B

Still down as of this morning.

NeilGunton

Quick update: The operating system was upgraded with some difficulties yesterday (Debian Lenny installed but then at the very last stage couldn't install grub or lilo onto the master boot record, so I eventually had to install the older Etch instead, followed by a dist-upgrade to Lenny). By last night I was able to copy most of the bulk of the data we have (the pics and database dumps) back to the server drives. Now today the task is to get all the various bits and pieces of software back up and running - DNS, subversion, sendmail, cron, mysql, replication, and the biggie - apache2.2, which I will probably be doing a custom build of rather than using the debian package (I need to configure a caching reverse proxy for performance).

So in a nutshell, it's not up yet, probably won't be until sometime tomorrow. Things are progressing, but since this is a major upgrade in terms of software, it's never just a simple case of restoring files from the backup.

Right now there are no showstoppers, it's just a question of cranking through it. Sorry for the inconvenience, this is a fairly infrequent occurrence, but the price we pay for being a one-man-shop with limited resources. Maybe one day we'll have the team of experts in their white lab coats and clipboards watching everything, three redundant load balancing hot spare servers distributed geographically and 99.999% uptime... for now ya just got me!

Neil

Neil_B

Congratulations Neil, you started the official update thread! The 'under service' page at CrazyGuy links to this thread. Neil F., again congratulation. Neil G., I hope the site's up soon. Best wishes, Neil B.

jamawani 

Has anyone seen Neil, lately?

RichardCranium

Yawn.... who cares?

jamawani 

RichardCranium? - Yawn, who cares?

Neil_B

Touring - it's a Neil thing.

KPH

Thanks for the updates!

acalek

Neil, hope you get everything back up with as little pain as possible, your site has been a few swedish newspapers, yours truly has been in the news here. peace

RichardCranium

Last I heard he was moving to NZ, and renouncing his US citizenship.

NeilGunton

I am making progress on the server rebuild, but as usual some little things seem to take a lot longer than they should. I am working on the webserver software itself now (apache), which is one of the last big steps. The database (mysql) is reloaded and replicating, dns is working, and I just got sendmail back up and working again (that is a real pain for some reason, very finicky about file permissions, and the config files seem to change with every release, which makes it very difficult to restore your previous config without breaking stuff).

In a nutshell - site is still down, there are no big problems at present but I still have to get apache2.2 working. I had quite a complex custom build setup with apache 1.3, so this isn't trivial, since the apache developers, in their wisdom, changed a lot of stuff between 1.3 and 2.2. I tried to do testing on my development workstation before doing the upgrade, but unfortunately couldn't complete it before my pre-scheduled trip to the datacenter. So I'm still working on it, and maybe we'll get something up tomorrow... but maybe not. I'll keep you posted here.

Thanks for your patience,

Neil

Tom Stormcrowe

Gotta love upgrades, right Neil?

Once done, though, all the hassles will be worth it.

acalek

Now the site has made it onto swedish tv as of tomorrow, Channel 4 has interviewed me. Neil keep on truckin and I agree once its all done it'll be worth it. from Sundsvall sweden, nacht

I'll post the link on the site as soon as its up

Goldrush

Go Neil go!

I'm having withdrawal symptoms here.

scruffyboy

I was getting nervous there. Thank you SO much for such a great site. It has been an inspiration for me. Don't know what I'd do without Crazy Guy On A Bike!

NeilGunton

Another update: I have been slowed down by 101 little problems building the newer versions of Apache, mod_perl and Embperl, and making it work in the configuration I want (caching reverse proxy, i.e. two custom builds of apache, pre-loading the Embperl code into shared memory). I have been making progress, but it's been maddeningly slow and frustrating, as always seems to be the case when you go off the "straight and narrow" with open source software. It's trivially easy to get Apache 2 working with the default debian setup, but that isn't a good setup for us. This is one of the reasons I put this off for so long - I've looked at doing this upgrade to Apache 2.x, mod_perl 2 and Embperl 2 for a while now (they've been out for years now), but I honestly thought they would be more stable and "finished" by now. Anyway, I seem to have finally been able to build everything.

Just to give one example: The latest problem was related to preloading the Embperl code so that it is shared between the Apache child processes... mod_perl built as a DSO (shared lib) wouldn't work, but building it statically into Apache produced some very strange compiler errors, which turned out to be related to the APR libs which conflicted with the ones from apt-get... so I de-installed those, which (for some reason) meant that subversion (the source control system) also went away since it depended on those libs. So then I had to go build subversion from source, which (for some reason) now didn't know about the dbd database which it was using previously, so then I had to go back and re-install subversion from apt-get, make a dump of it, de-install it again, build from source, rebuild the repository from the dump... it's a seemingly endless runaround of stupid tangents. Then there are all the little config and API changes that are new in "version 2" of the whole LAMP setup. Grrrr.

Anyway, the bottom line is this: There is progress, the webserver is now built and seems to be pre-loading the modules without errors, and so I can see the site again. However I am only opening it up to me at present, since there seem to still be some odd problems which I need to track down (e.g. when you click on 'Upload pic' on the edit page form, it doesn't do anything, but just puts you back on the same page). Obviously I have to go through all the functionality with a fine tooth comb and fix whatever subtle bugs have manifested with the new software. Also, I still need to install all the little cron scripts that do stuff like indexing for the search database, delivering email posts into the database and so on.

Incidentally, even after all of this is done, the really stupid thing about all this is that you, the end users, probably won't see any difference in the site at all. These upgrades are all "under the covers" and shouldn't affect the user-side functionality. That's what makes it so frustrating... very best case is that you end up with exactly what you had when you started. Sigh. But the underlying software and operating system should be better and more reliable in the long run, since they are more up-to-date and are actually being maintained and actively developed (unlike apache 1.3, which is very old now). For example, I was previously using some very old modules from Igor Sysoev over in Russia, for doing reverse proxy and content compression. This was necessary because mod_proxy in 1.3 was basically broken with respect to passing cookies in a reverse proxy setup. Now we are using Apache 2.2, I can go back to using the built in mod_proxy, since they seem to have fixed those problems now. It's better to be using up-to-date software.

Please bear with me, I know this has taken a lot longer than it should have, and I apologise to everybody who is out on the road and/or trying to use the site to keep in touch with their families. Unfortunately I really had to do this now as we are moving over to Oregon in August, and so I simply won't be able to have as easy physical access from over there (the server's in Chicago; I'm currently in St Louis, which is about 5 hours drive away).

Depending on how the bugs go, we might see the site back up sometime today, but I really do want to make sure I can do as much testing as possible before making it live to the public, so it might be tomorrow.

Thanks for your patience!

Neil

Tom2slow

we could so turn this into a distro debate, But I do feel your pain.....we use Centos at work and its the same Charlie Fox as Debian.

Speaking for everyone ( I assume)
We wish you luck!

And THANK YOU for the site....!!!!!!!!


Tom

jamawani 

Neil -

The finest pianists make what they perform sound easy.
That is the mark of genius.

Best - J

CCrew

IIS/.NET rulez

-Roger

BengeBoy 

FWIW, I have a new appreciation for the work Neil does on "behind the scenes" functionality that keeps the site running.

I was recently in South America traveling and was trying to use the web to keep up on the news, email my family, and make travel arrangements (my plans keep changing). Several internet cafes in small towns had pretty lousy service, and a number of transaction-heavy sites (like airline reservation sites and Expedia.com) would barely function at all (I finally gave up and used the phone to make airline reservations). In some places just pulling up a site like CNN.com was a struggle.

But I could always get to CGOAB, and it was always much faster than the others.

Thanks, Neil, for keeping the traveler in mind...

bb