crazyguyonabike down for server move
Unfortunately I have had to take crazyguyonabike offline in order to move the box to a new datacenter. Short story is that there is a problem with one of the RAID drives, and I was not getting the remote help I needed from the people at the datacenter in Las Vegas where the machine has been colocated for several years. This, combined with a series of mysterious, sporadic and unexplained network outages have finally made me decide to bite the bullet and just move the thing to be closer to where we are living, so that I can work on it myself when I need to. So, I have found an ISP here in Corvallis OR (Peak Internet) which seems reasonable. The big downer to all this is that I need to take the server completely down for a few days while it is shipped to me. I just took it down today, and the current datacenter people should be in the process of figuring out shipping, boxing up etc. I hope the machine will be with me sometime early next week, assuming we are able to do 2-day shipping without it being completely extortionate. I will then replace the failed hard drive, and install the server in the new datacenter. We are approaching capacity on the current set of hard drives, so I will need to buy new drives (8 of them, yikes) eventually, but I've decided to try to leave that until a bit later. For now I will focus on just getting the server to the new colo, fixing the drive, rebuilding the replacement, and getting things back online again asap. Then I will do the system rebuild with all new drives sometime down the line, when I am able to give users of the site more notice and everything can be planned.
I'm sorry I wasn't able to give much notice today, but things have not been moving quickly at all with the failed drive, and I have been frustrated that we couldn't just do this apparently simple thing (probably would have taken about 10 minutes if they could simply take the drives out one by one and look at serial numbers, since I had the S/N of the failed drive in a log file, but no, instead they insisted that I used a very obscure command line utility to make an LED light up on the front of the server so they could tell which drive it was - the catch being that I couldn't do this, since the drive had completely gone and so didn't even have a device number, which the command line utility needed, but telling them this was like talking to a brick wall). Anyway, the server has been running in a degraded state now (with the one drive down) since mid October, and if the other drive that was paired with the failed one also went down, then we would lose everything (it's RAID10, for anyone who's interested). Well, we wouldn't lose everything permanently, since I have MySQL replication both to a remote backup server in Germany and to my home workstation, and the pics are also replicated every minute, so worst case we would lose the last minute's worth of pic uploads if the server went down hard. But then I would need to rebuild the system from scratch, which is always a pain. I am keen to get it working properly again, and in the new datacenter, so once I had the new place lined up, I decided to just get it done asap rather than waiting around. I know this will be bad for some people who are on the road at the moment and trying to update their journals, but I just have to bite that bullet for a few days of downtime. If/when I get bigger then I can perhaps afford to buy hot spare servers, but I'm not remotely there yet. I don't want to use the backup server in Germany as a hot spare, because I am in the middle of moving house at the moment, so my own workstation may well be taken offline at any time to go to the new place. So, if the German server was the only one hosting the live website, it would have no backups. If it went down, we would potentially lose everything that had changed since the last snapshot, and I don't want to risk that. So, distasteful as it is, we have a few days of downtime, sorry about that. It doesn't happen very often, fwiw. Feel free to let me know I'm a crappy admin, unprofessional etc, I probably deserve it, but I'm just trying to fix what I regard as a critical problem as quickly as possible. Thanks for your patience, Neil |
Stuff Happens...
Feel free to let me know I'm a crappy admin, unprofessional etc, I probably deserve it, but I'm just trying to fix what I regard as a critical problem as quickly as possible. Aloha! Bob |
Originally Posted by statrixbob
(Post 14925895)
Nonsense. You do a great job providing a great service. Stuff happens from time to time and it sounds like you are dealing with it without the benefit of a lot of cooperation.
Aloha! Bob |
Thanks for the votes of confidence... the timing on all this could have been better, as we are also in the middle of moving house. In fact we just moved all our stuff a few days ago from Eureka CA up here to Corvallis OR. We thought we had found a good house, but it turned out not so great (won't go into details here), so we have been kind of up in the air the last few days, looking for another house. Add to this the fact that the rental market in this town is not great - it's a big college town, and the college has grown by leaps and bounds in recent years, with OSU seemingly unloading a lot of the accommodations load onto the town rental market. With so many students a lot of the available rentals are either really ratty and depressing, or else high end and out of our reach. So we're here in this temporary place, everything in cardboard boxes... then to have the website going down at the same time... it's a bit much. But that's life! You deal.
Update on the server: They told me it would cost $50 for ground, $150 for 2 day or $250 for overnight, so I opted for overnight. I don't know if that's a rip off, but I am really in no position to quibble over shipping costs. They're probably being straight about it but I have no way of telling, basically they have to ship it anyway so I have to get it done and worry about costs later. Apparently they missed the shipping window today (had to be by 1:30pm), so as I have already paid their invoice they should presumably be able to get it out the door tomorrow, Thursday... so assuming "overnight" really means what it says (and for $250 it should) then I might have the server here sometime on Friday. If it comes in the morning, which overnight to businesses often do (I think), then I might even be able to get the drive rebuilt and the server back up that same day. This is another reason I went for overnight - otherwise it would go out over the weekend, which is always a pain. Anyway, I'll keep you posted here and on twitter... Thanks again for your patience, Neil |
It is a great resource that charges nothing and only has a small bar to the side asking for donation, a few days down time is more than acceptable. Keep on keeping on and good luck with the move.
|
Originally Posted by chriskmurray
(Post 14926517)
It is a great resource that charges nothing and only has a small bar to the side asking for donation, a few days down time is more than acceptable. Keep on keeping on and good luck with the move.
Andrew |
Originally Posted by NeilGunton
(Post 14925151)
Feel free to let me know I'm a crappy admin, unprofessional etc, I probably deserve it, but I'm just trying to fix what I regard as a critical problem as quickly as possible.
Neil You give a service for which you do not charge (it's voluntary), no one can complain, you provide the touring community a great benefit. You have good hardware backup and replication and I think have done at low cost. You have also wrote the web server software yourself if I remember correctly. There is only yourself and you are clearly passionate about the site, even better you tour yourself. You are a combined engineer, site admin and web developer. A skillset that professionally would cost a lot to hire. The only point of failure has been the internet service provider, them not wanting to remove the drives, perhaps is from fear of being sued if something goes wrong (I'm not saying you would) just as a culture Americans from observing from the UK, like to sue. Such a situation can hamper companies being able to help, as possibly may be the cause in this case. I don't know not living there I would like third party opinion. You have given a clear explanation of the situation and feel bad about the server going down. You have a solution to the problem. You mention about the lack of planning to minimise server down time. Looking at it the server went into a degraded state mid october, right when you are moving house. The quickest solution is to get the drive replaced and plan a redeployment later. That is what you did, but the Internet Service provider were not playing ball, which is strange, would love to know why. Either way without a sufficient explanation from them it is better to move the server. The only way you could really have planned for such a situation is contingency planning and that takes big bucks so as a free service could not be done. So based on my clinical analysis there really is no fault on your part whatsever. cheers Steve Jubb |
Forgot to add also you are spending serious money to get the server moved. Another reason not your fault
|
Have you thought of issuing a redirect when people get to www.crazyguyonabike.com to a webpage just informing them?
If you need space I can help. Appreciate that may cause problems with the DNS updates for www.crazyguyonabike.com but your going to have update it anyway. |
I have REALLY enjoyed And benefited from Neil's site. This seems like a good time to consider a donation!!
|
Originally Posted by Tandem Tom
(Post 14926951)
I have REALLY enjoyed And benefited from Neil's site. This seems like a good time to consider a donation!!
|
Neil, thank you for the notice, and big thanks for all you do in providing such a wonderful site. It's time for me to make another donation!
|
Originally Posted by stephenjubb
(Post 14926818)
Have you thought of issuing a redirect when people get to www.crazyguyonabike.com to a webpage just informing them?
I think I can host it under my own plan (with DreamHost). |
I have used CGOAB to chronicle three tour journals so far. I have used CGOAB for research and entertainment for many years. It has proved invaluable in my trip planning; one of the very best sources for bicycling touring on the internet. I am pledging my donation ASAP.
|
I hope the moves go well Neil. That includes the server move and your moving house. I too will say that you do a great service to the bicycle touring community and I appreciate your efforts.
|
Thank goodness ,my morning was a mess , first "wiggo" getting hit by a car, then no cgoab to sooth my cycling fever but what to my surprise . Bike forums has the answer to one of my morning concerns.We must be patient , thanks Neil.
|
neil, down time is good, it allows you to keep your house in order. as a side note, i cant call you a crappy web host, no matter if you want me to or not :lol:
|
Another Corvallis biker
Neil, welcome to Corvallis! I live in Corvallis, let me know how I can help you with your move.
We use Peak at our business for internet connectivity, I highly recommend them. Dan |
Originally Posted by stephenjubb
(Post 14926818)
Have you thought of issuing a redirect when people get to www.crazyguyonabike.com to a webpage just informing them?
If you need space I can help. Appreciate that may cause problems with the DNS updates for www.crazyguyonabike.com but your going to have update it anyway. http://5.9.19.51/ The server has now been shipped, and UPS seems to confirm on the status page that it is being sent "Next Day Air", so I'm really hoping that all works as it should, and we actually see the server sometime tomorrow. Also hoping that everything will just go smoothly on the rebuild and installation in the new datacenter, but part of me wants to say "yeah, right, when has that ever gone smoothly"! I can think of many things that could go awry - e.g. UPS damages the server somehow in shipping, or the old datacenter forgot to include the spare drive, or even the server simply refuses to start up when I power it on. Oh well, no point worrying about stuff before it happens, I'll just keep fingers crossed and update here & on the website "maintenance" page as significant events occur. Thanks again for the good wishes, Neil |
hello welcome to corvallis. wonderful you are here. really great biking community you are becoming part of. why'd you choose Corvallis?
|
Neil,
Have you considered moving to the cloud? Pricing keeps dropping and no hardware hassles. Dave |
Originally Posted by sapote
(Post 14929165)
hello welcome to corvallis. wonderful you are here. really great biking community you are becoming part of. why'd you choose Corvallis?
Oops, I'm probably writing way too much off-topic stuff here, sorry about that. A lot's been happening and you're getting a kind of brain dump, my apologies. Neil |
Originally Posted by DAMO9966
(Post 14929182)
Neil,
Have you considered moving to the cloud? Pricing keeps dropping and no hardware hassles. Dave Neil |
It's all greek to me. All I know is you provide an awesome service and this is as good a spot as any to say THANKS AGAIN! for your site.
|
Besides the fact that I am going through withdrawal not getting my daily fix of journal reading... I can easily wait until the site is fixed:) good luck Neil, don't feel bad, you're doing awesome job and provide great service. People, consider donating or even subscribing. I subscribed monthly since this site beats pretty much any bike related reading.
Cloud? Just another word for bunch of servers sitting at a datacenter somewhere. How often Amazon Cloud services go down? If I had the choice of having my own server sitting at a location that I have access to I'd take that over any cloud. |
First of all, I'd like to express my support and thanks to Neil; thanks, man, as always!
And good luck and everything on your move to a new home and environment. I was only able to find out about the current problem when I got tired of unsuccessfully trying to access the website and finally googled something like "crazyguyonabike website problem" and landed on the "http://5.9.19.51/" page, and from there I clicked on to this bikeforum.net website (good excuse to join, too), so it would have been better to put a redirecting thing to the entire crazyguyonabike.com website address, but I guess that would take time to kick in, too. Anyway, glad to hear that things are under control. Maybe more people will decide to donate now. It's such a great website. Of course, I'm just another CGOAB addict! Thanks again! |
Welcome to the Willamette Valley, Neil.
Originally Posted by BigBlueToe
(Post 14927060)
Neil, thank you for the notice, and big thanks for all you do in providing such a wonderful site. It's time for me to make another donation!
If there were a link to a donation site posted, I could get that done right now. Otherwise, I'll be contributing once you're back up and running. |
Thanks for keeping us updated! I only discovered the site last week but it's had me hooked.
Have you got a temporary donation page or a paypal account so I can send you some money please. |
Originally Posted by NeilGunton
(Post 14925151)
Feel free to let me know I'm a crappy admin, unprofessional etc, I probably deserve it, but I'm just trying to fix what I regard as a critical problem as quickly as possible.
Just kidding! Thanks for Crazyguy, best of luck in the new home. |
A donation this red hot minute could be sent to Neil Gunton, Gen Del, Corvallis, OR 97333-9998. That might be quicker and more reliable than using his old address and trusting the forwarding process, seeing as he still has no address in Corvallis. Gen Del stuff is held a couple of weeks. If no pick up, it's returned to sender. Neil could call the post office to check for mail, thus saving him a wasted trip.
|
All times are GMT -6. The time now is 12:23 PM. |
Copyright © 2024 MH Sub I, LLC dba Internet Brands. All rights reserved. Use of this site indicates your consent to the Terms of Use.