Cycling and bicycle discussion forums. 
   Click here to join our community Log in to access your Control Panel  


Go Back   > >

Touring Have a dream to ride a bike across your state, across the country, or around the world? Self-contained or fully supported? Trade ideas, adventures, and more in our bicycle touring forum.

User Tag List

Reply
 
Thread Tools Search this Thread
Old 11-07-12, 04:16 PM   #1
NeilGunton
Crazyguyonabike
Thread Starter
 
Join Date: Nov 2003
Location: Albany, OR
Bikes: Co-Motion Divide
Posts: 684
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 29 Post(s)
crazyguyonabike down for server move

Unfortunately I have had to take crazyguyonabike offline in order to move the box to a new datacenter. Short story is that there is a problem with one of the RAID drives, and I was not getting the remote help I needed from the people at the datacenter in Las Vegas where the machine has been colocated for several years. This, combined with a series of mysterious, sporadic and unexplained network outages have finally made me decide to bite the bullet and just move the thing to be closer to where we are living, so that I can work on it myself when I need to. So, I have found an ISP here in Corvallis OR (Peak Internet) which seems reasonable. The big downer to all this is that I need to take the server completely down for a few days while it is shipped to me. I just took it down today, and the current datacenter people should be in the process of figuring out shipping, boxing up etc. I hope the machine will be with me sometime early next week, assuming we are able to do 2-day shipping without it being completely extortionate. I will then replace the failed hard drive, and install the server in the new datacenter. We are approaching capacity on the current set of hard drives, so I will need to buy new drives (8 of them, yikes) eventually, but I've decided to try to leave that until a bit later. For now I will focus on just getting the server to the new colo, fixing the drive, rebuilding the replacement, and getting things back online again asap. Then I will do the system rebuild with all new drives sometime down the line, when I am able to give users of the site more notice and everything can be planned.

I'm sorry I wasn't able to give much notice today, but things have not been moving quickly at all with the failed drive, and I have been frustrated that we couldn't just do this apparently simple thing (probably would have taken about 10 minutes if they could simply take the drives out one by one and look at serial numbers, since I had the S/N of the failed drive in a log file, but no, instead they insisted that I used a very obscure command line utility to make an LED light up on the front of the server so they could tell which drive it was - the catch being that I couldn't do this, since the drive had completely gone and so didn't even have a device number, which the command line utility needed, but telling them this was like talking to a brick wall). Anyway, the server has been running in a degraded state now (with the one drive down) since mid October, and if the other drive that was paired with the failed one also went down, then we would lose everything (it's RAID10, for anyone who's interested). Well, we wouldn't lose everything permanently, since I have MySQL replication both to a remote backup server in Germany and to my home workstation, and the pics are also replicated every minute, so worst case we would lose the last minute's worth of pic uploads if the server went down hard. But then I would need to rebuild the system from scratch, which is always a pain. I am keen to get it working properly again, and in the new datacenter, so once I had the new place lined up, I decided to just get it done asap rather than waiting around. I know this will be bad for some people who are on the road at the moment and trying to update their journals, but I just have to bite that bullet for a few days of downtime. If/when I get bigger then I can perhaps afford to buy hot spare servers, but I'm not remotely there yet. I don't want to use the backup server in Germany as a hot spare, because I am in the middle of moving house at the moment, so my own workstation may well be taken offline at any time to go to the new place. So, if the German server was the only one hosting the live website, it would have no backups. If it went down, we would potentially lose everything that had changed since the last snapshot, and I don't want to risk that. So, distasteful as it is, we have a few days of downtime, sorry about that. It doesn't happen very often, fwiw. Feel free to let me know I'm a crappy admin, unprofessional etc, I probably deserve it, but I'm just trying to fix what I regard as a critical problem as quickly as possible.

Thanks for your patience,

Neil
NeilGunton is offline   Reply With Quote
Old 11-07-12, 07:48 PM   #2
statrixbob
Newbie
 
Join Date: Jun 2008
Bikes:
Posts: 2
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Stuff Happens...

Quote:
Feel free to let me know I'm a crappy admin, unprofessional etc, I probably deserve it, but I'm just trying to fix what I regard as a critical problem as quickly as possible.
Nonsense. You do a great job providing a great service. Stuff happens from time to time and it sounds like you are dealing with it without the benefit of a lot of cooperation.

Aloha!

Bob
statrixbob is offline   Reply With Quote
Old 11-07-12, 08:30 PM   #3
mm718
Senior Member
 
Join Date: Sep 2011
Bikes:
Posts: 610
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
Quote:
Originally Posted by statrixbob View Post
Nonsense. You do a great job providing a great service. Stuff happens from time to time and it sounds like you are dealing with it without the benefit of a lot of cooperation.

Aloha!

Bob
+1
mm718 is offline   Reply With Quote
Old 11-07-12, 08:45 PM   #4
NeilGunton
Crazyguyonabike
Thread Starter
 
Join Date: Nov 2003
Location: Albany, OR
Bikes: Co-Motion Divide
Posts: 684
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 29 Post(s)
Thanks for the votes of confidence... the timing on all this could have been better, as we are also in the middle of moving house. In fact we just moved all our stuff a few days ago from Eureka CA up here to Corvallis OR. We thought we had found a good house, but it turned out not so great (won't go into details here), so we have been kind of up in the air the last few days, looking for another house. Add to this the fact that the rental market in this town is not great - it's a big college town, and the college has grown by leaps and bounds in recent years, with OSU seemingly unloading a lot of the accommodations load onto the town rental market. With so many students a lot of the available rentals are either really ratty and depressing, or else high end and out of our reach. So we're here in this temporary place, everything in cardboard boxes... then to have the website going down at the same time... it's a bit much. But that's life! You deal.

Update on the server: They told me it would cost $50 for ground, $150 for 2 day or $250 for overnight, so I opted for overnight. I don't know if that's a rip off, but I am really in no position to quibble over shipping costs. They're probably being straight about it but I have no way of telling, basically they have to ship it anyway so I have to get it done and worry about costs later. Apparently they missed the shipping window today (had to be by 1:30pm), so as I have already paid their invoice they should presumably be able to get it out the door tomorrow, Thursday... so assuming "overnight" really means what it says (and for $250 it should) then I might have the server here sometime on Friday. If it comes in the morning, which overnight to businesses often do (I think), then I might even be able to get the drive rebuilt and the server back up that same day. This is another reason I went for overnight - otherwise it would go out over the weekend, which is always a pain.

Anyway, I'll keep you posted here and on twitter...

Thanks again for your patience,

Neil
NeilGunton is offline   Reply With Quote
Old 11-08-12, 12:00 AM   #5
chriskmurray
Senior Member
 
chriskmurray's Avatar
 
Join Date: Aug 2012
Location: Colorado Springs
Bikes: Borealis Echo, Ground Up Designs Ti Cross bike, Xtracycle, GT mod trials bike, pixie race machine
Posts: 1,121
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
It is a great resource that charges nothing and only has a small bar to the side asking for donation, a few days down time is more than acceptable. Keep on keeping on and good luck with the move.
chriskmurray is offline   Reply With Quote
Old 11-08-12, 05:25 AM   #6
Aushiker
Senior Member
 
Aushiker's Avatar
 
Join Date: Jun 2007
Location: Fremantle, Western Australia
Bikes: Surly Long Haul Trucker, Salsa Mukluk, Giant Defy
Posts: 1,027
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 4 Post(s)
Quote:
Originally Posted by chriskmurray View Post
It is a great resource that charges nothing and only has a small bar to the side asking for donation, a few days down time is more than acceptable. Keep on keeping on and good luck with the move.
+ 1

Andrew
Aushiker is offline   Reply With Quote
Old 11-08-12, 05:57 AM   #7
stephenjubb
Senior Member
 
Join Date: Dec 2010
Bikes:
Posts: 91
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Quote:
Originally Posted by NeilGunton View Post
Feel free to let me know I'm a crappy admin, unprofessional etc, I probably deserve it, but I'm just trying to fix what I regard as a critical problem as quickly as possible.

Neil
As a Web Developer myself I am qualified to give opinion.
You give a service for which you do not charge (it's voluntary), no one can complain, you provide the touring community a great benefit.

You have good hardware backup and replication and I think have done at low cost. You have also wrote the web server software yourself if I remember correctly.

There is only yourself and you are clearly passionate about the site, even better you tour yourself. You are a combined engineer, site admin and web developer. A skillset that professionally would cost a lot to hire.

The only point of failure has been the internet service provider, them not wanting to remove the drives, perhaps is from fear of being sued if something goes wrong (I'm not saying you would) just as a culture Americans from observing from the UK, like to sue. Such a situation can hamper companies being able to help, as possibly may be the cause in this case. I don't know not living there I would like third party opinion.

You have given a clear explanation of the situation and feel bad about the server going down.

You have a solution to the problem.

You mention about the lack of planning to minimise server down time. Looking at it the server went into a degraded state mid october, right when you are moving house. The quickest solution is to get the drive replaced and plan a redeployment later.

That is what you did, but the Internet Service provider were not playing ball, which is strange, would love to know why. Either way without a sufficient explanation from them it is better to move the server.

The only way you could really have planned for such a situation is contingency planning and that takes big bucks so as a free service could not be done.

So based on my clinical analysis there really is no fault on your part whatsever.

cheers

Steve Jubb
stephenjubb is offline   Reply With Quote
Old 11-08-12, 05:58 AM   #8
stephenjubb
Senior Member
 
Join Date: Dec 2010
Bikes:
Posts: 91
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Forgot to add also you are spending serious money to get the server moved. Another reason not your fault
stephenjubb is offline   Reply With Quote
Old 11-08-12, 06:01 AM   #9
stephenjubb
Senior Member
 
Join Date: Dec 2010
Bikes:
Posts: 91
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Have you thought of issuing a redirect when people get to www.crazyguyonabike.com to a webpage just informing them?

If you need space I can help.

Appreciate that may cause problems with the DNS updates for www.crazyguyonabike.com but your going to have update it anyway.
stephenjubb is offline   Reply With Quote
Old 11-08-12, 07:11 AM   #10
Tandem Tom
Senior Member
 
Join Date: Dec 2011
Location: NE Ohio
Bikes: 1992 Serotta Colorado II,Co-Motion Speedster, Giant Escape Hybrid, 1977 Schwinn Super Le Tour
Posts: 1,304
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 7 Post(s)
I have REALLY enjoyed And benefited from Neil's site. This seems like a good time to consider a donation!!
Tandem Tom is offline   Reply With Quote
Old 11-08-12, 07:36 AM   #11
keenancook
Senior Member
 
Join Date: Jan 2012
Location: Fredericton, NB
Bikes: 2000 Specialized FSR, 2008 Cannondale T1
Posts: 53
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Quote:
Originally Posted by Tandem Tom View Post
I have REALLY enjoyed And benefited from Neil's site. This seems like a good time to consider a donation!!
+1 - I was thinking the same thing.
keenancook is offline   Reply With Quote
Old 11-08-12, 07:55 AM   #12
BigBlueToe
Senior Member
 
BigBlueToe's Avatar
 
Join Date: Aug 2006
Location: Central Coast, CA
Bikes: Surly LHT, Specialized Rockhopper, Nashbar Touring (old), Specialized Stumpjumper (older), Nishiki Tourer (model unknown)
Posts: 3,388
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Neil, thank you for the notice, and big thanks for all you do in providing such a wonderful site. It's time for me to make another donation!
BigBlueToe is offline   Reply With Quote
Old 11-08-12, 08:11 AM   #13
Al Cyone
Junior Member
 
Al Cyone's Avatar
 
Join Date: Nov 2012
Location: New York State
Bikes:
Posts: 7
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Quote:
Originally Posted by stephenjubb View Post
Have you thought of issuing a redirect when people get to www.crazyguyonabike.com to a webpage just informing them?
Or just an "Under Re-Construction" page with links to this forum?

I think I can host it under my own plan (with DreamHost).
Al Cyone is offline   Reply With Quote
Old 11-08-12, 10:24 AM   #14
Gus Riley
HomeBrew Master!
 
Gus Riley's Avatar
 
Join Date: Jan 2001
Location: West Central Illinois
Bikes: Aegis Aro Svelte, Surly LHT, Cannondal R3000 tandem, Santana Triplet.
Posts: 2,208
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
I have used CGOAB to chronicle three tour journals so far. I have used CGOAB for research and entertainment for many years. It has proved invaluable in my trip planning; one of the very best sources for bicycling touring on the internet. I am pledging my donation ASAP.
Gus Riley is offline   Reply With Quote
Old 11-08-12, 10:46 AM   #15
staehpj1 
Senior Member
 
staehpj1's Avatar
 
Join Date: Aug 2006
Location: Tallahassee, FL
Bikes:
Posts: 8,850
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 3 Post(s)
I hope the moves go well Neil. That includes the server move and your moving house. I too will say that you do a great service to the bicycle touring community and I appreciate your efforts.
staehpj1 is offline   Reply With Quote
Old 11-08-12, 11:25 AM   #16
lee kenney
Senior Member
 
Join Date: Dec 2010
Location: B.C.
Bikes: ritcheys{2** rm blizzard Geo elrick drop frame and acollection of parts bikes in waiting
Posts: 186
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Thank goodness ,my morning was a mess , first "wiggo" getting hit by a car, then no cgoab to sooth my cycling fever but what to my surprise . Bike forums has the answer to one of my morning concerns.We must be patient , thanks Neil.
lee kenney is offline   Reply With Quote
Old 11-08-12, 01:21 PM   #17
mr geeker
Day trip lover
 
mr geeker's Avatar
 
Join Date: May 2009
Location: capital city of iowa
Bikes: Giant Escape 3 (fair weather ride), Schwinn Frontier (winter/rain bike)
Posts: 756
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 4 Post(s)
neil, down time is good, it allows you to keep your house in order. as a side note, i cant call you a crappy web host, no matter if you want me to or not
mr geeker is offline   Reply With Quote
Old 11-08-12, 05:05 PM   #18
zan0416
Newbie
 
Join Date: Nov 2012
Location: Corvallis, OR
Bikes:
Posts: 1
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Another Corvallis biker

Neil, welcome to Corvallis! I live in Corvallis, let me know how I can help you with your move.

We use Peak at our business for internet connectivity, I highly recommend them.

Dan
zan0416 is offline   Reply With Quote
Old 11-08-12, 05:54 PM   #19
NeilGunton
Crazyguyonabike
Thread Starter
 
Join Date: Nov 2003
Location: Albany, OR
Bikes: Co-Motion Divide
Posts: 684
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 29 Post(s)
Quote:
Originally Posted by stephenjubb View Post
Have you thought of issuing a redirect when people get to www.crazyguyonabike.com to a webpage just informing them?

If you need space I can help.

Appreciate that may cause problems with the DNS updates for www.crazyguyonabike.com but your going to have update it anyway.
Thanks, I have redirected DNS to the backup server and cobbled together a temporary page which tries to explain what is going on. Unfortunately I'm having a problem getting it to work with https, but standard http links should work, like http://www.crazyguyonabike.com/. The DNS might take a while to propagate to some, since whatever DNS you use might still have the old address cached. If you want to see it directly, then you can go here:

http://5.9.19.51/

The server has now been shipped, and UPS seems to confirm on the status page that it is being sent "Next Day Air", so I'm really hoping that all works as it should, and we actually see the server sometime tomorrow. Also hoping that everything will just go smoothly on the rebuild and installation in the new datacenter, but part of me wants to say "yeah, right, when has that ever gone smoothly"! I can think of many things that could go awry - e.g. UPS damages the server somehow in shipping, or the old datacenter forgot to include the spare drive, or even the server simply refuses to start up when I power it on. Oh well, no point worrying about stuff before it happens, I'll just keep fingers crossed and update here & on the website "maintenance" page as significant events occur.

Thanks again for the good wishes,

Neil
NeilGunton is offline   Reply With Quote
Old 11-08-12, 05:56 PM   #20
sapote
Newbie
 
Join Date: Nov 2012
Bikes:
Posts: 3
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
hello welcome to corvallis. wonderful you are here. really great biking community you are becoming part of. why'd you choose Corvallis?
sapote is offline   Reply With Quote
Old 11-08-12, 06:02 PM   #21
DAMO9966
Newbie
 
Join Date: Nov 2012
Bikes:
Posts: 1
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Neil,

Have you considered moving to the cloud? Pricing keeps dropping and no hardware hassles.

Dave
DAMO9966 is offline   Reply With Quote
Old 11-08-12, 06:11 PM   #22
NeilGunton
Crazyguyonabike
Thread Starter
 
Join Date: Nov 2003
Location: Albany, OR
Bikes: Co-Motion Divide
Posts: 684
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 29 Post(s)
Quote:
Originally Posted by sapote View Post
hello welcome to corvallis. wonderful you are here. really great biking community you are becoming part of. why'd you choose Corvallis?
Thanks! My wife got a job offer here, and we've thought for a while that the place works well "on paper" in several ways, e.g. easy access to Portland, close to Eugene too, easy access to Oregon coast, temperate climate (ok lots of rain, but otherwise no extreme weather), no tsunamis, no hurricanes, no tornadoes, no extreme cold, no extreme heat, no wildfires. Ok, we're going to see a really big earthquake at some point when the Cascadia Subduction Zone finally ruptures again, but I think we'll be in better shape here than we would have been in Eureka, which is right near the Mendocino Triple Junction (where the San Andreas meets up with the Mendocino Fault and the Cascadia Subduction Zone - one of the most geologically active places in the country, and overdue for an 8+ earthquake). And there is the risk of occasional flooding from the river, but we're not in the flood plain as far as I know. And we have access to all the natural wonders of Oregon, and it's a very bike friendly city, and with the students (always pros and cons with them) we get a town that has a fair amount of culture and amenities. We can drive up to the major urban center easily (90 minutes to Portland on I5 for Chiho's Japanese grocery store). Finding an ok rental house has been hard, but just today we finalized our second attempt (first attempt we thought would be good, but there are, um, some issues with the landlady which I won't go into here). The new house is on Princess, just off Walnut. So there is that busy road very close by, but at least we're not right on it. And the house has some crucial aspects for our two cats: A sliding door in the back (so I can put in one of those patio door inserts with a cat flap to give them access to outside, which they need since they were both adopted as adult strays - they just go nuts if they can't go out) and the back yard, while small, also has ready-made holes in the fences so they can get out and explore (the rear has access to the greenway). So hopefully things are working out - they even gave us 1/2 off on the first two months full rent! Yay, it's been really expensive moving up here so we could use a break. And the server is on its way, so here's hoping things are finally turning around.

Oops, I'm probably writing way too much off-topic stuff here, sorry about that. A lot's been happening and you're getting a kind of brain dump, my apologies.

Neil
NeilGunton is offline   Reply With Quote
Old 11-08-12, 06:22 PM   #23
NeilGunton
Crazyguyonabike
Thread Starter
 
Join Date: Nov 2003
Location: Albany, OR
Bikes: Co-Motion Divide
Posts: 684
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 29 Post(s)
Quote:
Originally Posted by DAMO9966 View Post
Neil,

Have you considered moving to the cloud? Pricing keeps dropping and no hardware hassles.

Dave
Yeah, that's an obvious question given these hardware issues. I don't have a fully rational response to that, except to say that I still don't really trust the cloud concept. I guess maybe I'm just old school, but I like to have full control over the server, down to the metal. I like that I can spec a really nice RAID10 setup with 8 drives, giving me very predictable IO performance. I also like that I'm not at the mercy of random outages in what are extremely complex, multi-layered systems that I have no control over. My site is of a size that can very well be hosted on a single box, in fact this box can do a lot more without breaking a sweat. The problem with this downtime is really unusual, having to bring everything to a screeching halt just so I can send the server to a different physical datacenter seems like an argument against hosting your own hardware, but it really doesn't happen very often. Also, remember that there have been big outages with all the cloud providers lately, so nothing is immune to downtime. At least my downtime is under my control, and I know how to fix it. Then there's just the simple security of knowing exactly where your data is. But of course there are arguments to be made the other way too - as I said, it's not a fully rational thing, more like an intuitive preference for being in full control right down to the hardware. I do feel like I actually get better performance this way, with everything running on a single local box, rather than relying on all those abstraction layers and distributed services that AWS etc entail. I know that when I've used websites that are hosted on the cloud, they often just don't seem as responsive and "zippy" as one that is hosted on a dedicated server.

Neil

Last edited by NeilGunton; 11-08-12 at 07:08 PM. Reason: speling and grammer
NeilGunton is offline   Reply With Quote
Old 11-08-12, 06:36 PM   #24
reed523
Senior Member
 
Join Date: Oct 2005
Location: Perkins, Oklahoma
Bikes: A few
Posts: 237
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
It's all greek to me. All I know is you provide an awesome service and this is as good a spot as any to say THANKS AGAIN! for your site.
reed523 is offline   Reply With Quote
Old 11-08-12, 08:44 PM   #25
AdamDZ 
Bike addict, dreamer
 
AdamDZ's Avatar
 
Join Date: Nov 2009
Location: Queens, New York
Bikes:
Posts: 5,165
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Besides the fact that I am going through withdrawal not getting my daily fix of journal reading... I can easily wait until the site is fixed good luck Neil, don't feel bad, you're doing awesome job and provide great service. People, consider donating or even subscribing. I subscribed monthly since this site beats pretty much any bike related reading.

Cloud? Just another word for bunch of servers sitting at a datacenter somewhere. How often Amazon Cloud services go down? If I had the choice of having my own server sitting at a location that I have access to I'd take that over any cloud.
AdamDZ is offline   Reply With Quote
Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



All times are GMT -6. The time now is 02:58 AM.