crazyguyonabike down for server move
#1
Crazyguyonabike
Thread Starter
Join Date: Nov 2003
Location: Lebanon, OR
Posts: 697
Bikes: Co-Motion Divide
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 35 Post(s)
Likes: 0
Liked 3 Times
in
2 Posts
crazyguyonabike down for server move
Unfortunately I have had to take crazyguyonabike offline in order to move the box to a new datacenter. Short story is that there is a problem with one of the RAID drives, and I was not getting the remote help I needed from the people at the datacenter in Las Vegas where the machine has been colocated for several years. This, combined with a series of mysterious, sporadic and unexplained network outages have finally made me decide to bite the bullet and just move the thing to be closer to where we are living, so that I can work on it myself when I need to. So, I have found an ISP here in Corvallis OR (Peak Internet) which seems reasonable. The big downer to all this is that I need to take the server completely down for a few days while it is shipped to me. I just took it down today, and the current datacenter people should be in the process of figuring out shipping, boxing up etc. I hope the machine will be with me sometime early next week, assuming we are able to do 2-day shipping without it being completely extortionate. I will then replace the failed hard drive, and install the server in the new datacenter. We are approaching capacity on the current set of hard drives, so I will need to buy new drives (8 of them, yikes) eventually, but I've decided to try to leave that until a bit later. For now I will focus on just getting the server to the new colo, fixing the drive, rebuilding the replacement, and getting things back online again asap. Then I will do the system rebuild with all new drives sometime down the line, when I am able to give users of the site more notice and everything can be planned.
I'm sorry I wasn't able to give much notice today, but things have not been moving quickly at all with the failed drive, and I have been frustrated that we couldn't just do this apparently simple thing (probably would have taken about 10 minutes if they could simply take the drives out one by one and look at serial numbers, since I had the S/N of the failed drive in a log file, but no, instead they insisted that I used a very obscure command line utility to make an LED light up on the front of the server so they could tell which drive it was - the catch being that I couldn't do this, since the drive had completely gone and so didn't even have a device number, which the command line utility needed, but telling them this was like talking to a brick wall). Anyway, the server has been running in a degraded state now (with the one drive down) since mid October, and if the other drive that was paired with the failed one also went down, then we would lose everything (it's RAID10, for anyone who's interested). Well, we wouldn't lose everything permanently, since I have MySQL replication both to a remote backup server in Germany and to my home workstation, and the pics are also replicated every minute, so worst case we would lose the last minute's worth of pic uploads if the server went down hard. But then I would need to rebuild the system from scratch, which is always a pain. I am keen to get it working properly again, and in the new datacenter, so once I had the new place lined up, I decided to just get it done asap rather than waiting around. I know this will be bad for some people who are on the road at the moment and trying to update their journals, but I just have to bite that bullet for a few days of downtime. If/when I get bigger then I can perhaps afford to buy hot spare servers, but I'm not remotely there yet. I don't want to use the backup server in Germany as a hot spare, because I am in the middle of moving house at the moment, so my own workstation may well be taken offline at any time to go to the new place. So, if the German server was the only one hosting the live website, it would have no backups. If it went down, we would potentially lose everything that had changed since the last snapshot, and I don't want to risk that. So, distasteful as it is, we have a few days of downtime, sorry about that. It doesn't happen very often, fwiw. Feel free to let me know I'm a crappy admin, unprofessional etc, I probably deserve it, but I'm just trying to fix what I regard as a critical problem as quickly as possible.
Thanks for your patience,
Neil
I'm sorry I wasn't able to give much notice today, but things have not been moving quickly at all with the failed drive, and I have been frustrated that we couldn't just do this apparently simple thing (probably would have taken about 10 minutes if they could simply take the drives out one by one and look at serial numbers, since I had the S/N of the failed drive in a log file, but no, instead they insisted that I used a very obscure command line utility to make an LED light up on the front of the server so they could tell which drive it was - the catch being that I couldn't do this, since the drive had completely gone and so didn't even have a device number, which the command line utility needed, but telling them this was like talking to a brick wall). Anyway, the server has been running in a degraded state now (with the one drive down) since mid October, and if the other drive that was paired with the failed one also went down, then we would lose everything (it's RAID10, for anyone who's interested). Well, we wouldn't lose everything permanently, since I have MySQL replication both to a remote backup server in Germany and to my home workstation, and the pics are also replicated every minute, so worst case we would lose the last minute's worth of pic uploads if the server went down hard. But then I would need to rebuild the system from scratch, which is always a pain. I am keen to get it working properly again, and in the new datacenter, so once I had the new place lined up, I decided to just get it done asap rather than waiting around. I know this will be bad for some people who are on the road at the moment and trying to update their journals, but I just have to bite that bullet for a few days of downtime. If/when I get bigger then I can perhaps afford to buy hot spare servers, but I'm not remotely there yet. I don't want to use the backup server in Germany as a hot spare, because I am in the middle of moving house at the moment, so my own workstation may well be taken offline at any time to go to the new place. So, if the German server was the only one hosting the live website, it would have no backups. If it went down, we would potentially lose everything that had changed since the last snapshot, and I don't want to risk that. So, distasteful as it is, we have a few days of downtime, sorry about that. It doesn't happen very often, fwiw. Feel free to let me know I'm a crappy admin, unprofessional etc, I probably deserve it, but I'm just trying to fix what I regard as a critical problem as quickly as possible.
Thanks for your patience,
Neil
#2
Newbie
Join Date: Jun 2008
Posts: 2
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Likes: 0
Liked 0 Times
in
0 Posts
Stuff Happens...
Feel free to let me know I'm a crappy admin, unprofessional etc, I probably deserve it, but I'm just trying to fix what I regard as a critical problem as quickly as possible.
Aloha!
Bob
#3
Senior Member
Join Date: Sep 2011
Posts: 626
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 8 Post(s)
Likes: 0
Liked 0 Times
in
0 Posts
#4
Crazyguyonabike
Thread Starter
Join Date: Nov 2003
Location: Lebanon, OR
Posts: 697
Bikes: Co-Motion Divide
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 35 Post(s)
Likes: 0
Liked 3 Times
in
2 Posts
Thanks for the votes of confidence... the timing on all this could have been better, as we are also in the middle of moving house. In fact we just moved all our stuff a few days ago from Eureka CA up here to Corvallis OR. We thought we had found a good house, but it turned out not so great (won't go into details here), so we have been kind of up in the air the last few days, looking for another house. Add to this the fact that the rental market in this town is not great - it's a big college town, and the college has grown by leaps and bounds in recent years, with OSU seemingly unloading a lot of the accommodations load onto the town rental market. With so many students a lot of the available rentals are either really ratty and depressing, or else high end and out of our reach. So we're here in this temporary place, everything in cardboard boxes... then to have the website going down at the same time... it's a bit much. But that's life! You deal.
Update on the server: They told me it would cost $50 for ground, $150 for 2 day or $250 for overnight, so I opted for overnight. I don't know if that's a rip off, but I am really in no position to quibble over shipping costs. They're probably being straight about it but I have no way of telling, basically they have to ship it anyway so I have to get it done and worry about costs later. Apparently they missed the shipping window today (had to be by 1:30pm), so as I have already paid their invoice they should presumably be able to get it out the door tomorrow, Thursday... so assuming "overnight" really means what it says (and for $250 it should) then I might have the server here sometime on Friday. If it comes in the morning, which overnight to businesses often do (I think), then I might even be able to get the drive rebuilt and the server back up that same day. This is another reason I went for overnight - otherwise it would go out over the weekend, which is always a pain.
Anyway, I'll keep you posted here and on twitter...
Thanks again for your patience,
Neil
Update on the server: They told me it would cost $50 for ground, $150 for 2 day or $250 for overnight, so I opted for overnight. I don't know if that's a rip off, but I am really in no position to quibble over shipping costs. They're probably being straight about it but I have no way of telling, basically they have to ship it anyway so I have to get it done and worry about costs later. Apparently they missed the shipping window today (had to be by 1:30pm), so as I have already paid their invoice they should presumably be able to get it out the door tomorrow, Thursday... so assuming "overnight" really means what it says (and for $250 it should) then I might have the server here sometime on Friday. If it comes in the morning, which overnight to businesses often do (I think), then I might even be able to get the drive rebuilt and the server back up that same day. This is another reason I went for overnight - otherwise it would go out over the weekend, which is always a pain.
Anyway, I'll keep you posted here and on twitter...
Thanks again for your patience,
Neil
#5
Senior Member
Join Date: Aug 2012
Location: Colorado Springs
Posts: 1,134
Bikes: Borealis Echo, Ground Up Designs Ti Cross bike, Xtracycle, GT mod trials bike, pixie race machine
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 8 Post(s)
Likes: 0
Liked 2 Times
in
2 Posts
It is a great resource that charges nothing and only has a small bar to the side asking for donation, a few days down time is more than acceptable. Keep on keeping on and good luck with the move.
#7
Senior Member
Join Date: Dec 2010
Posts: 91
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Likes: 0
Liked 0 Times
in
0 Posts
You give a service for which you do not charge (it's voluntary), no one can complain, you provide the touring community a great benefit.
You have good hardware backup and replication and I think have done at low cost. You have also wrote the web server software yourself if I remember correctly.
There is only yourself and you are clearly passionate about the site, even better you tour yourself. You are a combined engineer, site admin and web developer. A skillset that professionally would cost a lot to hire.
The only point of failure has been the internet service provider, them not wanting to remove the drives, perhaps is from fear of being sued if something goes wrong (I'm not saying you would) just as a culture Americans from observing from the UK, like to sue. Such a situation can hamper companies being able to help, as possibly may be the cause in this case. I don't know not living there I would like third party opinion.
You have given a clear explanation of the situation and feel bad about the server going down.
You have a solution to the problem.
You mention about the lack of planning to minimise server down time. Looking at it the server went into a degraded state mid october, right when you are moving house. The quickest solution is to get the drive replaced and plan a redeployment later.
That is what you did, but the Internet Service provider were not playing ball, which is strange, would love to know why. Either way without a sufficient explanation from them it is better to move the server.
The only way you could really have planned for such a situation is contingency planning and that takes big bucks so as a free service could not be done.
So based on my clinical analysis there really is no fault on your part whatsever.
cheers
Steve Jubb
#9
Senior Member
Join Date: Dec 2010
Posts: 91
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Likes: 0
Liked 0 Times
in
0 Posts
Have you thought of issuing a redirect when people get to www.crazyguyonabike.com to a webpage just informing them?
If you need space I can help.
Appreciate that may cause problems with the DNS updates for www.crazyguyonabike.com but your going to have update it anyway.
If you need space I can help.
Appreciate that may cause problems with the DNS updates for www.crazyguyonabike.com but your going to have update it anyway.
#10
Senior Member
Join Date: Dec 2011
Location: NE Ohio
Posts: 2,596
Bikes: 1992 Serotta Colorado II,Co-Motion Speedster, Giant Escape Hybrid, 1977 Schwinn Super Le Tour
Mentioned: 4 Post(s)
Tagged: 0 Thread(s)
Quoted: 457 Post(s)
Likes: 0
Liked 112 Times
in
85 Posts
I have REALLY enjoyed And benefited from Neil's site. This seems like a good time to consider a donation!!
#11
Senior Member
Join Date: Jan 2012
Location: Fredericton, NB
Posts: 53
Bikes: 2000 Specialized FSR, 2008 Cannondale T1
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Likes: 0
Liked 0 Times
in
0 Posts
#12
Senior Member
Join Date: Aug 2006
Location: Central Coast, CA
Posts: 3,392
Bikes: Surly LHT, Specialized Rockhopper, Nashbar Touring (old), Specialized Stumpjumper (older), Nishiki Tourer (model unknown)
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
Likes: 0
Liked 2 Times
in
2 Posts
Neil, thank you for the notice, and big thanks for all you do in providing such a wonderful site. It's time for me to make another donation!
#13
Junior Member
Join Date: Nov 2012
Location: New York State
Posts: 8
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Likes: 0
Liked 0 Times
in
0 Posts
Have you thought of issuing a redirect when people get to www.crazyguyonabike.com to a webpage just informing them?
I think I can host it under my own plan (with DreamHost).
#14
HomeBrew Master!
Join Date: Jan 2001
Location: West Central Illinois
Posts: 2,208
Bikes: Aegis Aro Svelte, Surly LHT, Cannondal R3000 tandem, Santana Triplet.
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Likes: 0
Liked 0 Times
in
0 Posts
I have used CGOAB to chronicle three tour journals so far. I have used CGOAB for research and entertainment for many years. It has proved invaluable in my trip planning; one of the very best sources for bicycling touring on the internet. I am pledging my donation ASAP.
#15
Senior Member
I hope the moves go well Neil. That includes the server move and your moving house. I too will say that you do a great service to the bicycle touring community and I appreciate your efforts.
#16
Senior Member
Join Date: Dec 2010
Location: B.C.
Posts: 189
Bikes: ritcheys{2** rm blizzard Geo elrick drop frame and acollection of parts bikes in waiting
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 2 Post(s)
Likes: 0
Liked 0 Times
in
0 Posts
Thank goodness ,my morning was a mess , first "wiggo" getting hit by a car, then no cgoab to sooth my cycling fever but what to my surprise . Bike forums has the answer to one of my morning concerns.We must be patient , thanks Neil.
#17
Day trip lover
Join Date: May 2009
Location: capital city of iowa
Posts: 813
Bikes: '16 Giant Escape 3 (fair weather ride), Giant Quasar (work in progress), 2002 saturn vue (crap weather ride)
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 21 Post(s)
Likes: 0
Liked 0 Times
in
0 Posts
neil, down time is good, it allows you to keep your house in order. as a side note, i cant call you a crappy web host, no matter if you want me to or not
#18
Newbie
Join Date: Nov 2012
Location: Corvallis, OR
Posts: 1
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Likes: 0
Liked 0 Times
in
0 Posts
Another Corvallis biker
Neil, welcome to Corvallis! I live in Corvallis, let me know how I can help you with your move.
We use Peak at our business for internet connectivity, I highly recommend them.
Dan
We use Peak at our business for internet connectivity, I highly recommend them.
Dan
#19
Crazyguyonabike
Thread Starter
Join Date: Nov 2003
Location: Lebanon, OR
Posts: 697
Bikes: Co-Motion Divide
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 35 Post(s)
Likes: 0
Liked 3 Times
in
2 Posts
Have you thought of issuing a redirect when people get to www.crazyguyonabike.com to a webpage just informing them?
If you need space I can help.
Appreciate that may cause problems with the DNS updates for www.crazyguyonabike.com but your going to have update it anyway.
If you need space I can help.
Appreciate that may cause problems with the DNS updates for www.crazyguyonabike.com but your going to have update it anyway.
https://5.9.19.51/
The server has now been shipped, and UPS seems to confirm on the status page that it is being sent "Next Day Air", so I'm really hoping that all works as it should, and we actually see the server sometime tomorrow. Also hoping that everything will just go smoothly on the rebuild and installation in the new datacenter, but part of me wants to say "yeah, right, when has that ever gone smoothly"! I can think of many things that could go awry - e.g. UPS damages the server somehow in shipping, or the old datacenter forgot to include the spare drive, or even the server simply refuses to start up when I power it on. Oh well, no point worrying about stuff before it happens, I'll just keep fingers crossed and update here & on the website "maintenance" page as significant events occur.
Thanks again for the good wishes,
Neil
#22
Crazyguyonabike
Thread Starter
Join Date: Nov 2003
Location: Lebanon, OR
Posts: 697
Bikes: Co-Motion Divide
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 35 Post(s)
Likes: 0
Liked 3 Times
in
2 Posts
Oops, I'm probably writing way too much off-topic stuff here, sorry about that. A lot's been happening and you're getting a kind of brain dump, my apologies.
Neil
#23
Crazyguyonabike
Thread Starter
Join Date: Nov 2003
Location: Lebanon, OR
Posts: 697
Bikes: Co-Motion Divide
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 35 Post(s)
Likes: 0
Liked 3 Times
in
2 Posts
Neil
Last edited by NeilGunton; 11-08-12 at 07:08 PM. Reason: speling and grammer
#24
Full Member
Join Date: Oct 2005
Location: Perkins, Oklahoma
Posts: 241
Bikes: A few
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 2 Post(s)
Likes: 0
Liked 0 Times
in
0 Posts
It's all greek to me. All I know is you provide an awesome service and this is as good a spot as any to say THANKS AGAIN! for your site.
#25
Bike addict, dreamer
Join Date: Nov 2009
Location: Queens, New York
Posts: 5,165
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 4 Post(s)
Likes: 0
Liked 0 Times
in
0 Posts
Besides the fact that I am going through withdrawal not getting my daily fix of journal reading... I can easily wait until the site is fixed good luck Neil, don't feel bad, you're doing awesome job and provide great service. People, consider donating or even subscribing. I subscribed monthly since this site beats pretty much any bike related reading.
Cloud? Just another word for bunch of servers sitting at a datacenter somewhere. How often Amazon Cloud services go down? If I had the choice of having my own server sitting at a location that I have access to I'd take that over any cloud.
Cloud? Just another word for bunch of servers sitting at a datacenter somewhere. How often Amazon Cloud services go down? If I had the choice of having my own server sitting at a location that I have access to I'd take that over any cloud.