Advertise on Bikeforums.net



User Tag List

Page 1 of 4 123 ... LastLast
Results 1 to 25 of 99
  1. #1
    Crazyguyonabike
    Join Date
    Nov 2003
    Location
    Albany, OR
    My Bikes
    Co-Motion Divide
    Posts
    585
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    crazyguyonabike down for server move

    Unfortunately I have had to take crazyguyonabike offline in order to move the box to a new datacenter. Short story is that there is a problem with one of the RAID drives, and I was not getting the remote help I needed from the people at the datacenter in Las Vegas where the machine has been colocated for several years. This, combined with a series of mysterious, sporadic and unexplained network outages have finally made me decide to bite the bullet and just move the thing to be closer to where we are living, so that I can work on it myself when I need to. So, I have found an ISP here in Corvallis OR (Peak Internet) which seems reasonable. The big downer to all this is that I need to take the server completely down for a few days while it is shipped to me. I just took it down today, and the current datacenter people should be in the process of figuring out shipping, boxing up etc. I hope the machine will be with me sometime early next week, assuming we are able to do 2-day shipping without it being completely extortionate. I will then replace the failed hard drive, and install the server in the new datacenter. We are approaching capacity on the current set of hard drives, so I will need to buy new drives (8 of them, yikes) eventually, but I've decided to try to leave that until a bit later. For now I will focus on just getting the server to the new colo, fixing the drive, rebuilding the replacement, and getting things back online again asap. Then I will do the system rebuild with all new drives sometime down the line, when I am able to give users of the site more notice and everything can be planned.

    I'm sorry I wasn't able to give much notice today, but things have not been moving quickly at all with the failed drive, and I have been frustrated that we couldn't just do this apparently simple thing (probably would have taken about 10 minutes if they could simply take the drives out one by one and look at serial numbers, since I had the S/N of the failed drive in a log file, but no, instead they insisted that I used a very obscure command line utility to make an LED light up on the front of the server so they could tell which drive it was - the catch being that I couldn't do this, since the drive had completely gone and so didn't even have a device number, which the command line utility needed, but telling them this was like talking to a brick wall). Anyway, the server has been running in a degraded state now (with the one drive down) since mid October, and if the other drive that was paired with the failed one also went down, then we would lose everything (it's RAID10, for anyone who's interested). Well, we wouldn't lose everything permanently, since I have MySQL replication both to a remote backup server in Germany and to my home workstation, and the pics are also replicated every minute, so worst case we would lose the last minute's worth of pic uploads if the server went down hard. But then I would need to rebuild the system from scratch, which is always a pain. I am keen to get it working properly again, and in the new datacenter, so once I had the new place lined up, I decided to just get it done asap rather than waiting around. I know this will be bad for some people who are on the road at the moment and trying to update their journals, but I just have to bite that bullet for a few days of downtime. If/when I get bigger then I can perhaps afford to buy hot spare servers, but I'm not remotely there yet. I don't want to use the backup server in Germany as a hot spare, because I am in the middle of moving house at the moment, so my own workstation may well be taken offline at any time to go to the new place. So, if the German server was the only one hosting the live website, it would have no backups. If it went down, we would potentially lose everything that had changed since the last snapshot, and I don't want to risk that. So, distasteful as it is, we have a few days of downtime, sorry about that. It doesn't happen very often, fwiw. Feel free to let me know I'm a crappy admin, unprofessional etc, I probably deserve it, but I'm just trying to fix what I regard as a critical problem as quickly as possible.

    Thanks for your patience,

    Neil

  2. #2
    Newbie
    Join Date
    Jun 2008
    Posts
    2
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Stuff Happens...

    Feel free to let me know I'm a crappy admin, unprofessional etc, I probably deserve it, but I'm just trying to fix what I regard as a critical problem as quickly as possible.
    Nonsense. You do a great job providing a great service. Stuff happens from time to time and it sounds like you are dealing with it without the benefit of a lot of cooperation.

    Aloha!

    Bob

  3. #3
    Senior Member
    Join Date
    Sep 2011
    Posts
    353
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by statrixbob View Post
    Nonsense. You do a great job providing a great service. Stuff happens from time to time and it sounds like you are dealing with it without the benefit of a lot of cooperation.

    Aloha!

    Bob
    +1

  4. #4
    Crazyguyonabike
    Join Date
    Nov 2003
    Location
    Albany, OR
    My Bikes
    Co-Motion Divide
    Posts
    585
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Thanks for the votes of confidence... the timing on all this could have been better, as we are also in the middle of moving house. In fact we just moved all our stuff a few days ago from Eureka CA up here to Corvallis OR. We thought we had found a good house, but it turned out not so great (won't go into details here), so we have been kind of up in the air the last few days, looking for another house. Add to this the fact that the rental market in this town is not great - it's a big college town, and the college has grown by leaps and bounds in recent years, with OSU seemingly unloading a lot of the accommodations load onto the town rental market. With so many students a lot of the available rentals are either really ratty and depressing, or else high end and out of our reach. So we're here in this temporary place, everything in cardboard boxes... then to have the website going down at the same time... it's a bit much. But that's life! You deal.

    Update on the server: They told me it would cost $50 for ground, $150 for 2 day or $250 for overnight, so I opted for overnight. I don't know if that's a rip off, but I am really in no position to quibble over shipping costs. They're probably being straight about it but I have no way of telling, basically they have to ship it anyway so I have to get it done and worry about costs later. Apparently they missed the shipping window today (had to be by 1:30pm), so as I have already paid their invoice they should presumably be able to get it out the door tomorrow, Thursday... so assuming "overnight" really means what it says (and for $250 it should) then I might have the server here sometime on Friday. If it comes in the morning, which overnight to businesses often do (I think), then I might even be able to get the drive rebuilt and the server back up that same day. This is another reason I went for overnight - otherwise it would go out over the weekend, which is always a pain.

    Anyway, I'll keep you posted here and on twitter...

    Thanks again for your patience,

    Neil

  5. #5
    Senior Member chriskmurray's Avatar
    Join Date
    Aug 2012
    Location
    Colorado Springs
    My Bikes
    Borealis Echo, Ground Up Designs Ti Cross bike, Xtracycle, GT mod trials bike, pixie race machine
    Posts
    858
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    It is a great resource that charges nothing and only has a small bar to the side asking for donation, a few days down time is more than acceptable. Keep on keeping on and good luck with the move.

  6. #6
    Senior Member Aushiker's Avatar
    Join Date
    Jun 2007
    Posts
    978
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by chriskmurray View Post
    It is a great resource that charges nothing and only has a small bar to the side asking for donation, a few days down time is more than acceptable. Keep on keeping on and good luck with the move.
    + 1

    Andrew

  7. #7
    Senior Member
    Join Date
    Dec 2010
    Posts
    89
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by NeilGunton View Post
    Feel free to let me know I'm a crappy admin, unprofessional etc, I probably deserve it, but I'm just trying to fix what I regard as a critical problem as quickly as possible.

    Neil
    As a Web Developer myself I am qualified to give opinion.
    You give a service for which you do not charge (it's voluntary), no one can complain, you provide the touring community a great benefit.

    You have good hardware backup and replication and I think have done at low cost. You have also wrote the web server software yourself if I remember correctly.

    There is only yourself and you are clearly passionate about the site, even better you tour yourself. You are a combined engineer, site admin and web developer. A skillset that professionally would cost a lot to hire.

    The only point of failure has been the internet service provider, them not wanting to remove the drives, perhaps is from fear of being sued if something goes wrong (I'm not saying you would) just as a culture Americans from observing from the UK, like to sue. Such a situation can hamper companies being able to help, as possibly may be the cause in this case. I don't know not living there I would like third party opinion.

    You have given a clear explanation of the situation and feel bad about the server going down.

    You have a solution to the problem.

    You mention about the lack of planning to minimise server down time. Looking at it the server went into a degraded state mid october, right when you are moving house. The quickest solution is to get the drive replaced and plan a redeployment later.

    That is what you did, but the Internet Service provider were not playing ball, which is strange, would love to know why. Either way without a sufficient explanation from them it is better to move the server.

    The only way you could really have planned for such a situation is contingency planning and that takes big bucks so as a free service could not be done.

    So based on my clinical analysis there really is no fault on your part whatsever.

    cheers

    Steve Jubb

  8. #8
    Senior Member
    Join Date
    Dec 2010
    Posts
    89
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Forgot to add also you are spending serious money to get the server moved. Another reason not your fault

  9. #9
    Senior Member
    Join Date
    Dec 2010
    Posts
    89
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Have you thought of issuing a redirect when people get to www.crazyguyonabike.com to a webpage just informing them?

    If you need space I can help.

    Appreciate that may cause problems with the DNS updates for www.crazyguyonabike.com but your going to have update it anyway.

  10. #10
    Senior Member
    Join Date
    Dec 2011
    Location
    NE Ohio
    My Bikes
    1992 Serotta Colorado II,Co-Motion Speedster, Giant Escape Hybrid, 1977 Schwinn Super Le Tour
    Posts
    770
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I have REALLY enjoyed And benefited from Neil's site. This seems like a good time to consider a donation!!

  11. #11
    Senior Member
    Join Date
    Jan 2012
    Location
    Fredericton, NB
    My Bikes
    2000 Specialized FSR, 2008 Cannondale T1
    Posts
    53
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by Tandem Tom View Post
    I have REALLY enjoyed And benefited from Neil's site. This seems like a good time to consider a donation!!
    +1 - I was thinking the same thing.

  12. #12
    Senior Member BigBlueToe's Avatar
    Join Date
    Aug 2006
    Location
    Central Coast, CA
    My Bikes
    Surly LHT, Specialized Rockhopper, Nashbar Touring (old), Specialized Stumpjumper (older), Nishiki Tourer (model unknown)
    Posts
    3,388
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Neil, thank you for the notice, and big thanks for all you do in providing such a wonderful site. It's time for me to make another donation!

  13. #13
    Junior Member Al Cyone's Avatar
    Join Date
    Nov 2012
    Location
    New York State
    Posts
    8
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by stephenjubb View Post
    Have you thought of issuing a redirect when people get to www.crazyguyonabike.com to a webpage just informing them?
    Or just an "Under Re-Construction" page with links to this forum?

    I think I can host it under my own plan (with DreamHost).

  14. #14
    HomeBrew Master! Gus Riley's Avatar
    Join Date
    Jan 2001
    Location
    West Central Illinois
    My Bikes
    Aegis Aro Svelte, Surly LHT, Cannondal R3000 tandem, Santana Triplet.
    Posts
    2,208
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I have used CGOAB to chronicle three tour journals so far. I have used CGOAB for research and entertainment for many years. It has proved invaluable in my trip planning; one of the very best sources for bicycling touring on the internet. I am pledging my donation ASAP.
    2012 TransAm Tour journal link: http://www.crazyguyonabike.com/doc/Threeisacharm

    Naked Carbon Weave Aegis Aro Svelte, Purpleen Cannondale RT3000 Tandem, Orange Santana Triplet, Surly Long Haul Trucker

    So little pains do the vulgar take in the investigation of truth, accepting readily the first story that comes to hand." Thucydides, 4th Century B.C.E.



    http://tickers.TickerFactory.com/ezt...33f/weight.png

  15. #15
    Senior Member staehpj1's Avatar
    Join Date
    Aug 2006
    Location
    Parkville, Md
    Posts
    7,615
    Mentioned
    3 Post(s)
    Tagged
    0 Thread(s)
    I hope the moves go well Neil. That includes the server move and your moving house. I too will say that you do a great service to the bicycle touring community and I appreciate your efforts.

  16. #16
    Senior Member
    Join Date
    Dec 2010
    Location
    B.C.
    My Bikes
    ritcheys{2** rm blizzard Geo elrick drop frame and acollection of parts bikes in waiting
    Posts
    178
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Thank goodness ,my morning was a mess , first "wiggo" getting hit by a car, then no cgoab to sooth my cycling fever but what to my surprise . Bike forums has the answer to one of my morning concerns.We must be patient , thanks Neil.

  17. #17
    Day trip lover mr geeker's Avatar
    Join Date
    May 2009
    Location
    some where in iowa
    My Bikes
    trek 830
    Posts
    541
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    neil, down time is good, it allows you to keep your house in order. as a side note, i cant call you a crappy web host, no matter if you want me to or not
    instant human: just add coffee
    trek 830 mountain track - dead

  18. #18
    Newbie
    Join Date
    Nov 2012
    Location
    Corvallis, OR
    Posts
    1
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Another Corvallis biker

    Neil, welcome to Corvallis! I live in Corvallis, let me know how I can help you with your move.

    We use Peak at our business for internet connectivity, I highly recommend them.

    Dan

  19. #19
    Crazyguyonabike
    Join Date
    Nov 2003
    Location
    Albany, OR
    My Bikes
    Co-Motion Divide
    Posts
    585
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by stephenjubb View Post
    Have you thought of issuing a redirect when people get to www.crazyguyonabike.com to a webpage just informing them?

    If you need space I can help.

    Appreciate that may cause problems with the DNS updates for www.crazyguyonabike.com but your going to have update it anyway.
    Thanks, I have redirected DNS to the backup server and cobbled together a temporary page which tries to explain what is going on. Unfortunately I'm having a problem getting it to work with https, but standard http links should work, like http://www.crazyguyonabike.com/. The DNS might take a while to propagate to some, since whatever DNS you use might still have the old address cached. If you want to see it directly, then you can go here:

    http://5.9.19.51/

    The server has now been shipped, and UPS seems to confirm on the status page that it is being sent "Next Day Air", so I'm really hoping that all works as it should, and we actually see the server sometime tomorrow. Also hoping that everything will just go smoothly on the rebuild and installation in the new datacenter, but part of me wants to say "yeah, right, when has that ever gone smoothly"! I can think of many things that could go awry - e.g. UPS damages the server somehow in shipping, or the old datacenter forgot to include the spare drive, or even the server simply refuses to start up when I power it on. Oh well, no point worrying about stuff before it happens, I'll just keep fingers crossed and update here & on the website "maintenance" page as significant events occur.

    Thanks again for the good wishes,

    Neil

  20. #20
    Newbie
    Join Date
    Nov 2012
    Posts
    3
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    hello welcome to corvallis. wonderful you are here. really great biking community you are becoming part of. why'd you choose Corvallis?

  21. #21
    Newbie
    Join Date
    Nov 2012
    Posts
    1
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Neil,

    Have you considered moving to the cloud? Pricing keeps dropping and no hardware hassles.

    Dave

  22. #22
    Crazyguyonabike
    Join Date
    Nov 2003
    Location
    Albany, OR
    My Bikes
    Co-Motion Divide
    Posts
    585
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by sapote View Post
    hello welcome to corvallis. wonderful you are here. really great biking community you are becoming part of. why'd you choose Corvallis?
    Thanks! My wife got a job offer here, and we've thought for a while that the place works well "on paper" in several ways, e.g. easy access to Portland, close to Eugene too, easy access to Oregon coast, temperate climate (ok lots of rain, but otherwise no extreme weather), no tsunamis, no hurricanes, no tornadoes, no extreme cold, no extreme heat, no wildfires. Ok, we're going to see a really big earthquake at some point when the Cascadia Subduction Zone finally ruptures again, but I think we'll be in better shape here than we would have been in Eureka, which is right near the Mendocino Triple Junction (where the San Andreas meets up with the Mendocino Fault and the Cascadia Subduction Zone - one of the most geologically active places in the country, and overdue for an 8+ earthquake). And there is the risk of occasional flooding from the river, but we're not in the flood plain as far as I know. And we have access to all the natural wonders of Oregon, and it's a very bike friendly city, and with the students (always pros and cons with them) we get a town that has a fair amount of culture and amenities. We can drive up to the major urban center easily (90 minutes to Portland on I5 for Chiho's Japanese grocery store). Finding an ok rental house has been hard, but just today we finalized our second attempt (first attempt we thought would be good, but there are, um, some issues with the landlady which I won't go into here). The new house is on Princess, just off Walnut. So there is that busy road very close by, but at least we're not right on it. And the house has some crucial aspects for our two cats: A sliding door in the back (so I can put in one of those patio door inserts with a cat flap to give them access to outside, which they need since they were both adopted as adult strays - they just go nuts if they can't go out) and the back yard, while small, also has ready-made holes in the fences so they can get out and explore (the rear has access to the greenway). So hopefully things are working out - they even gave us 1/2 off on the first two months full rent! Yay, it's been really expensive moving up here so we could use a break. And the server is on its way, so here's hoping things are finally turning around.

    Oops, I'm probably writing way too much off-topic stuff here, sorry about that. A lot's been happening and you're getting a kind of brain dump, my apologies.

    Neil

  23. #23
    Crazyguyonabike
    Join Date
    Nov 2003
    Location
    Albany, OR
    My Bikes
    Co-Motion Divide
    Posts
    585
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by DAMO9966 View Post
    Neil,

    Have you considered moving to the cloud? Pricing keeps dropping and no hardware hassles.

    Dave
    Yeah, that's an obvious question given these hardware issues. I don't have a fully rational response to that, except to say that I still don't really trust the cloud concept. I guess maybe I'm just old school, but I like to have full control over the server, down to the metal. I like that I can spec a really nice RAID10 setup with 8 drives, giving me very predictable IO performance. I also like that I'm not at the mercy of random outages in what are extremely complex, multi-layered systems that I have no control over. My site is of a size that can very well be hosted on a single box, in fact this box can do a lot more without breaking a sweat. The problem with this downtime is really unusual, having to bring everything to a screeching halt just so I can send the server to a different physical datacenter seems like an argument against hosting your own hardware, but it really doesn't happen very often. Also, remember that there have been big outages with all the cloud providers lately, so nothing is immune to downtime. At least my downtime is under my control, and I know how to fix it. Then there's just the simple security of knowing exactly where your data is. But of course there are arguments to be made the other way too - as I said, it's not a fully rational thing, more like an intuitive preference for being in full control right down to the hardware. I do feel like I actually get better performance this way, with everything running on a single local box, rather than relying on all those abstraction layers and distributed services that AWS etc entail. I know that when I've used websites that are hosted on the cloud, they often just don't seem as responsive and "zippy" as one that is hosted on a dedicated server.

    Neil
    Last edited by NeilGunton; 11-08-12 at 06:08 PM. Reason: speling and grammer

  24. #24
    Senior Member
    Join Date
    Oct 2005
    Location
    Perkins, Oklahoma
    My Bikes
    Salsa Fargo is my current love
    Posts
    210
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    It's all greek to me. All I know is you provide an awesome service and this is as good a spot as any to say THANKS AGAIN! for your site.

  25. #25
    Bike addict, dreamer AdamDZ's Avatar
    Join Date
    Nov 2009
    Location
    Queens, New York
    My Bikes
    Surly Disc trucker (DIY), Fuji Reveal 1.0 (DIY MTB), Specialized Roubaix
    Posts
    5,161
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    Besides the fact that I am going through withdrawal not getting my daily fix of journal reading... I can easily wait until the site is fixed good luck Neil, don't feel bad, you're doing awesome job and provide great service. People, consider donating or even subscribing. I subscribed monthly since this site beats pretty much any bike related reading.

    Cloud? Just another word for bunch of servers sitting at a datacenter somewhere. How often Amazon Cloud services go down? If I had the choice of having my own server sitting at a location that I have access to I'd take that over any cloud.

Page 1 of 4 123 ... LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •