HostGeekZ

The forums are now back online and should be working correctly for the most part.

There does appear to be some missing threads towards the end of the table which we are unable to recover due to the way it has repaired.

Whilst repairing the table the database server kernel paniced which corrupted the thread and user tables. The temporary data files were so corrupt that they infact were reading 16tb in disk usage(incorrect), we have recovered these and the only things that appear to be missing is some newer threads, instead of restoring a backup from yesterday this seems the best course of action to sustain.

However due to the consistant kernel panics the database server encounters we are deciding on what to do about it, we do however have the kernel panic and have repaired the main cause of that now.

Apologises for the extended downtime however we could not have prevented it.

-Scott

Siu Blue Wind

Scott, I did not understand a word you just said but thank you to you and Joe anyway.

Stacey

He said that the kernel (Colonel?) is prone to anexiety attacks and needs the help of a good psycartrist

DannoXYZ 

Good job guys! I think we should line up an alternate supplier of chickens so that the Colonel doesn't panic when the supply gets low...

linux_author

https://www.linux-ha.org/

RoadToad

huh?

Air

Someone feed that man a chicken!

Great job guys, thannks!

Fritz_S

Any hope at all that more of the lost recent threads will be restored? I am a junior member, so 1/2 of my subscribed threads, and 3/4 of my postings are GONE! I may never forget how to ride a bike, but I might forget (for example) where that advice on fitting a saddle went. (whine, ..whinge, ..., whimper, ...) -- but THANKS to the heroic IT folks for all they could save!!

Machka 

Newer threads .... like everything in the last MONTH or so? It was like some weird Twilight Zone flashback coming on here today!! Can any of it be restored?

Stacey

Want some



with that whine?

HostGeekZ

Hello,

It does appear more than just a few threads from the bottom of the table are missing as originally thought. Due to the options available being leave it as it presently is, roll back to yesterday(loosing a full day of posting, which would also mean another 8 hours just to repair) it's best just to leave it like it is.

Unfortunately all missing threads will not be returning and will have to be re-created, we are looking into possibly inserting specific data from those threads to newly created ones.

This is essentially what happened, we were reparing the database(because it was not showing the post body, this is due to the table being corrupt), in the middle of repairing the threads table the kernel paniced which caused the server to halt in the middle of repairing the data files, when it was back online the thread table was reporting a 127 error(MySQL error code 127: Record-file is crashed), we recovered this as much as we could as the usual mysql repairing tools(myisamchk,mysqlcheck) are not designed to recover in such a way and all they done was loop over and over until the disk was full. After recovering everything we could from the TMD file we made this the new data file(hence some missing threads) and then began to repair the index to have the id's resort in order.

What all of that means is, the database crashed while being repaired, that broke the threads completely, we recovered rather than restoring a backup(loosing a day worth of data), at first it appeared only a few threads were missing but there does appear to be more than the initial inspection however now it's not worth going back to well over 24 hours worth of data being missing and another 8 hours of downtime to repair this.

All we can do is apologize for the inconvenience and we are looking into making sure this doesn't happen again, starting with a new database server to resolve the constant problems the present one has.

-Scott

webist

Does Joe have enough cash reserves to accomplish this? Or, do we need to have a fundraiser?

banerjek

As good luck would have it, topics tend to be revisited frequently on BF anyway....

Katzenjammer

For them wot don't know what "kernal panic" means, it's a term from a particular computing subculture meaning that the central part of the software that runs the server got completely confused, stopped working (which is not unlike "stopped juggling the chainsaws"), and screamed for help.

A public-service announcement from a retired nerd.

Katzenjammer

I hope someone is still working hard on the forum software, btw. I've noticed that the paging is broken...you won't necessarily see all the posts in a thread unless you diddle around with the posts-per-page option.

Brian

The server's farked. Carked it.