Southern California - open letter to Internet Brands on behalf of SoCal BF

In light of the quoted message below, an e-mail has been sent to Ryan pointing him to this thread.


We aren't here to monkey around with a formula that has proven to work for years already. BikeForums is what it is because of what you all have made it, and if we were to come in and start screwing with the place, thereby driving more and more of you away, then this site will have lost all its value, both to you as a user and to us as the parent company. Hopefully, we can improve this site from the back end (tech and development support) and bring you guys a better BikeForums experience.

Please don't hesitate to shoot me a PM or an email if you have any questions, and I'll see you on the boards!

Ryan
rfujitani@internetbrands.com

Dear Sir/Madam:

The last two nights, BikeForums has suffered multi-hour outages. Outages are frequent, and even when the forum is operating, sluggish performance has been the norm in recent days. This is not acceptable. For an example of how the server outages and performance degradations threaten the forum, I point you to the Southern California regional subforum's regular members. Southern California represents a huge contingent of BikeForums, with many highly active and visible members who liven up not only the regional subforum but also many non-region specific subforums. This demographic also represents a lucrative market that your advertisers covet. (A recent informal survey shows that members of this group spent well over $100k in the purchase of new bicycles--never mind related purchases of a similar magnitude--so far in 2007.) I am a member of this group and also a moderator on the forum at large.

Many of us spend several hours a day on the forum, but if the forum cannot be rendered stable, you may see a migration of some of this community to another forum. At this point in time, I am not advocating this, but it is a logical alternative to the frustrations we have experienced in recent days with this forum, and if a critical mass of this community goes, I most definitely will go with them. You would not only lose a major contingent of users, but you would also lose a lot of positive free exposure. The Southern California BikeForums users are some of your strongest advocates, with three jerseys promoting Southern California BikeForums (all produced at our own expense) which we proudly wear at virtually every major organized ride in the region.

The forum needs to be made stable now. BikeForums as a mere presence on the Internet is worth essentially nothing. It is the community that gives this site value, but this community is not captive. I have not been impressed with the recent handling of the forum by its new owners. Clearly, many changes made have been made without sufficient testing, and the promised improved back end support is far from adequate. (In particular, how any outage can last for several hours is beyond my comprehension.) I do not know what changes need to be made, be it a code rollback, removal of complex ad insertions, or something else, but something must be done to dramatically improve forum performance.

I look forward to a response, but more importantly, know that I will be watching how the site behaves. I like it here, but my patience is being tested.


Respectfully,
Jason Chen

Thanks, Jschen, for sending that out. I sure hope they don't have a contract on uptime, otherwise they'd be kicked out a long time ago. I'm not a database guy, but perhaps they are just running on old hardware, and they don't have the budget to purchase appropriate hardware to scale to the capacities that the user groups demand. Any database, that gets too many hits, and can't handle the requests is going to suffer like we have seen. However, I do agree ... with the promise of improved backend support, it is actually really piss poor.

I ran into a similar issue at work a few years ago, because there was a bug in the software, when under heavy load, there would be a memory leak. There was nothing the vendor would do, because the software was out of date, and they only real support they could give was with a new upgrade, which was a complete software architecture change. No matter what memory upgrade we did, it didn't would. In fact, we even had a dual CPU server, and it didn't make it any better. It required changing OS (from Windows server to Solaris), and changing major software version to rectify the problem. We ended up changing server platform, and changing software vendor. It was a long 2 year project (merging multiple KB databases).

life is hard.

Thank you Jason....
Very well said.....I hate to think of this going down right before a ride....bad news...
If my clients systems acted this way...I would be fired....
At the least they should be notifiying us of changes, issues,etc...
I dont really want to migrate away.....but if they dont get this fixed then I see no alternative....

To paraphrase Dire StraitsI WANT MY BF

maybe they could fix it with one of these (www.violin-memory.com)

Every time I start to go through withdrawals when the forum is down, I realize that I am an internet addict and need help.

I'd really hate to see this forum go away, but things change and as we are seeing, it's not always for the best. I hope they care enough to keep BF going.

Thanks, Jason.

Very nice letter, Jason.

Thank you Jason for putting into words what most of us have been feeling in recent days.

Thanks Jason.

For me I just wanted to know the reason for outages and perhaps the plan to address them. Hmm... everything I type at work feels so work-like. :p

[EDIT] I didn't mean to sound harsh or demanding. In any case, thanks for maintaining this free BikeForums.net site.

Thanks, Jason, for letting them know. :)

Hi Everyone,

I just wanted to respond here and let you know that we've received the notice. I'm going to go ahead and ask one of our tech guys to step in and address the issue. Thanks for making your grievances known in a civil manner; you'd be surprised how rare that is in online communities these days.

Ryan

Hi Everyone,

I just wanted to respond here and let you know that we've received the notice. I'm going to go ahead and ask one of our tech guys to step in and address the issue. Thanks for making your grievances known in a civil manner; you'd be surprised how rare that is in online communities these days.

Ryan

Cool!

Well, Ryan, we actually are a very civil group regardless of our occassional dramas, IMHO. Most folks here joke about being addicted to BF when in reality, we are addicted to talking to each other and riding our bikes, often together. This forum provides us a venue to connect easily - so the better it operates, the better it is for all of us. We really WANT it work. :)

Thanks, Ryan.

Hi Everyone,

I just wanted to respond here and let you know that we've received the notice. I'm going to go ahead and ask one of our tech guys to step in and address the issue. Thanks for making your grievances known in a civil manner; you'd be surprised how rare that is in online communities these days.

Ryan



Shoot, Ryan - if you are indeed located in LA like it says you are, come on out and join us for a ride or meal!!! ;)

Thanks for responding.

Well, Ryan, we actually are a very civil group regardless of our occassional dramas, IMHO. Most folks here joke about being addicted to BF when in reality, we are addicted to talking to each other and riding our bikes, often together. This forum provides us a venue to connect easily - so the better it operates, the better it is for all of us. We really WANT it work. :)

Yeah, I totally get what you're saying. BF is a very civil community, and it's become one of my favorites. I very much want it to work as well, and I know that it starts with actual functionality. Can't have a community if there's no place to meet! Sorry about all the craziness lately; I'm not qualified to speak on those things, unfortunately, so I have to rely on my tech guys for that. Thanks for your patience! :)

Shoot, Ryan - if you are indeed located in LA like it says you are, come on out and join us for a ride or meal!!! ;)

Thanks for responding.

Haha, Jason's been asking me that since I came on the forum. I seriously should, shouldn't I?? :D

This is Nate from the Internet Brands tech team.

First of all, I apologize for the recent outages on the forums. I personally have always been supportive of addicts, and as a matter of principle I believe that it's important to continue supplying addicts with opportunities to engage in the behavior mandated by their addictions. Unfortunately, sometimes there are interruptions, and these are unpleasant for all of us.

The reason for the recent BikeForums outages seems to be related to either recurring hardware failures or to a misbehaving operating system. Either way, the solution is going to be to migrate the site to completely new servers. We've ordered new machines for the site, which will be installed at the data center that hosts the majority of our Internet Brands sites, including CarsDirect.com, Autos.com, WikiTravel.org, etc. In addition to solving our outage issues, these servers should represent a performance upgrade for the site. Migrating the site will also make it easier for us to do performance tuning on the MySQL database, which should make the site quicker and more responsive as well.

The downside here is that this process is probably going to take about a month, which means that there will be occasional outages until we've been able to get set up shop on the new servers. We're going to be as vigilant as possible (with the help of the team at AdminGeekZ, who are currently helping us with server admin) about tackling these outages as soon as we're aware of them.

We hope that you'll continue to be patient as we iron out these issues. You all are a great community, and we're committed to doing everything we can to keep BikeForums running as smoothly as possible.

Thanks,

Nate

With the main issues now in the open, I would also like to highlight a secondary issue with forum performance. The forum's moderators are unpaid volunteers, donating their time to help the forum operate smoothly. With poor server performance, even simple tasks often take an inordinate amount of time, and tracking user posting histories or other more involved tasks are often out of the question. I cannot speak for other moderators, but when server performance goes downhill, my moderating efforts are met only with frustration, and in the last week, I have simply given up on achieving any moderator tasks (no matter how simple) when the forum is not performing well. Without robust hardware and software infrastructure, other elements of the BikeForums infrastructure may quickly fall into disrepair, too.

Ryan, thank you for bringing this to the attention of the appropriate personal, and Nate, thank you for your clarification about the situation.

Ryan,

put it on a mainframe. . .

Could put it on a load balancer. I seriously doubt that it's the database having troubles (MySQL can go through A LOT of stress), so it would only mean another web server. Two webservers are nothing for big sites too anyway, especially running clunky vBulletin, and I seriously doubt we're running two being as we have errors so often and they happen to everyone. And with how many ads are on this site, you could pay for a new webserver with the extra money obtained from simply having more uptime.


However, being from a forum community that I was on for some 6 years, with reoccurring problems like this until it turned into like a 2 month downtime. Do NOT let the techs screw with you (if they're the hosting's techs), if they wont fix it within an extremely short period, drop this hosting package as soon as you find another one. Saving $10 a month isn't worth 2 month downtimes. You'll end up losing people to more stable sites. Especially considering this isn't a hard setup to duplicate. :p

ah c'mon...can't the little hamsters just get a rest now and then?

I mean, what if it was you or one of your loved one's in that tiny wheel...:D

For what it's worth, Nate is actually pretty awesome. Anytime a problem on BF is brought to my attention, I tell him and he takes care of it right away. Plus, he looks like he should be the Norse god of heavy metal and has a wicked sense of humor. Thank you, Nate, for being righteous, for taking care of my BF troubles, and for rocking out in Valhalla. :D

first world problems:

my ipod broke,
my bmw got a flat,
bikeforums is running slow.

hey guys it was me that caused the outage with my glorious posts.

Sorry!

Thanks, Ryan, Nate and the rest of the tech team! :)

Plus, he looks like he should be the Norse god of heavy metal and has a wicked sense of humor. :D



How about posting some pics?? :p

first world problems:

my ipod broke,
my bmw got a flat,
bikeforums is running slow.



Way better than:

My ditch that I call a toilet sprung a leak.
My water buffalo got a flat.
The capitalists stole our land.

How about posting some pics?? :p
:p

I would if I had any.
Then again, if he found out, he might be inclined to destroy me and ride over my carcass in his vikingmobile.