Database error.
03-15-25 | 06:17 AM
#101
Not lost, just exploring
Joined: Jun 2019
Posts: 1,607
Likes: 1,757
From: Near the Heart of OH
Bikes: '25 Jamis Renegade S1, '18 Quick 1,'04 Trek 2300, '97 730 Multitrack, '95 750 Multitrack, and a few others
NO problems here. I have visited the site on both mobile and desktop browsers. Currently posting from Firefox on Ubuntu linux. Just fired up an old Macbook with Safari and it's working as expected.
03-17-25 | 11:25 AM
#102
Should Be More Popular
Joined: Dec 2007
Posts: 46,115
Likes: 11,718
From: Malvern, PA (20 miles West of Philly)
Bikes: 1986 Alpine (steel road bike), 2009 Ti Habenero, 2013 Specialized Roubaix
03-17-25 | 06:23 PM
#103
Highly Enriched Driftium
Joined: Apr 2017
Posts: 6,667
Likes: 2,156
I'm just a user, not admin.
Just a brief summary from early in this thread, for those new to it:
- This is not system-specific. Not your computer or phone, not operating system, not browser. It's not you.
- This database error is caused by the Bike Forum servers being overwhelmed by "web crawlers", which "scrape data" from websites. It's like if every BF user, or many more than that, were all simultaneously flipping through every page on every thread, in order to read every post, all at once. In years past, the motive might have been to get data for targeted advertising and other things. Now, it seems to be powered by companies/entities/countries competing in Artificial Intelligence, which is now driven by "Large Language Models", which attempt to read every single thing published on the internet, and due to the feverish competition, do it frequently to build their database as quickly as possible. These data searches are supposed to be "polite", in a way that does not crash servers. Of late, they are not. The techs at BF have tried to detect this occurance, and put in a time delay or blocking a website, but it is difficult. I am guessing that web crawlers can easily change or disguise their identity and origin, making it difficult to permanently block a particular one.
- The BF servers had been down for many hours in the past, but in recent months, seems to reboot(?) in a matter of minutes when an error occurs. Not perfect, but way better than it was.
- Permanent resolution of this problem may require (only my guess), enforceable rules/legislation that would make these crawlers self-identify, follow the rules of the road, respect refusals, etc. That's way, WAY above my knowledge and understanding of the web, not my profession. If I can remember, next time I see someone I know who is retired from high up in computer tech, and is still active in trade associations, I'll ask if they know answers/suggestions on this.
Just a brief summary from early in this thread, for those new to it:
- This is not system-specific. Not your computer or phone, not operating system, not browser. It's not you.
- This database error is caused by the Bike Forum servers being overwhelmed by "web crawlers", which "scrape data" from websites. It's like if every BF user, or many more than that, were all simultaneously flipping through every page on every thread, in order to read every post, all at once. In years past, the motive might have been to get data for targeted advertising and other things. Now, it seems to be powered by companies/entities/countries competing in Artificial Intelligence, which is now driven by "Large Language Models", which attempt to read every single thing published on the internet, and due to the feverish competition, do it frequently to build their database as quickly as possible. These data searches are supposed to be "polite", in a way that does not crash servers. Of late, they are not. The techs at BF have tried to detect this occurance, and put in a time delay or blocking a website, but it is difficult. I am guessing that web crawlers can easily change or disguise their identity and origin, making it difficult to permanently block a particular one.
- The BF servers had been down for many hours in the past, but in recent months, seems to reboot(?) in a matter of minutes when an error occurs. Not perfect, but way better than it was.
- Permanent resolution of this problem may require (only my guess), enforceable rules/legislation that would make these crawlers self-identify, follow the rules of the road, respect refusals, etc. That's way, WAY above my knowledge and understanding of the web, not my profession. If I can remember, next time I see someone I know who is retired from high up in computer tech, and is still active in trade associations, I'll ask if they know answers/suggestions on this.
Last edited by Duragrouch; 03-17-25 at 08:32 PM.
03-17-25 | 10:54 PM
#104
Senior Member
Joined: Apr 2019
Posts: 3,822
Likes: 1,450
From: UK
It should not need a reboot to recover from this if that is indeed what’s happening. The service provider, the web server they host, or the forum engine itself (worst option) should detect this and have some kind of flood / ddos control to protect the resources of the application and decline those connections. I’m assuming that the forum isn’t self hosted (I help run one which is and you get a lot more control and lower ongoing costs then but also have to solve all your own problems 
You don’t need legislation and likely won’t get global agreement and compliance. Distributed denial of service attacks have been doing this for decades deliberately and there are known defences (which do need adaptation over time)
You don’t need legislation and likely won’t get global agreement and compliance. Distributed denial of service attacks have been doing this for decades deliberately and there are known defences (which do need adaptation over time)
03-17-25 | 11:42 PM
#105
Highly Enriched Driftium
Joined: Apr 2017
Posts: 6,667
Likes: 2,156
(above) Thank you, it does seem you know infinitely more about this than me. I only guessed a reboot because of the resolution being non instant, but also not that long (I think perhaps 10 minutes or less). But the techs improved something, because it had been hours before.
Due to LLM-AI and increasing computing power devoted to it, I think this type of thing is only going to get worse, unless good countermeasures are found. Looking online only now, I can see this is a hot topic of discussion quite recently. Since it's a problem for virtually all on the open web, I'm hoping that will drive demand for a solution. Here's a good article (for us layfolks) with summarized bullet points, not TL: DR.
https://medium.com/ai-frontiers/ai-c...t-04caebace08e
Due to LLM-AI and increasing computing power devoted to it, I think this type of thing is only going to get worse, unless good countermeasures are found. Looking online only now, I can see this is a hot topic of discussion quite recently. Since it's a problem for virtually all on the open web, I'm hoping that will drive demand for a solution. Here's a good article (for us layfolks) with summarized bullet points, not TL: DR.
https://medium.com/ai-frontiers/ai-c...t-04caebace08e
03-18-25 | 05:33 AM
#106
Should Be More Popular
Joined: Dec 2007
Posts: 46,115
Likes: 11,718
From: Malvern, PA (20 miles West of Philly)
Bikes: 1986 Alpine (steel road bike), 2009 Ti Habenero, 2013 Specialized Roubaix
I'm just a user, not admin.
Just a brief summary from early in this thread, for those new to it:
- This is not system-specific. Not your computer or phone, not operating system, not browser. It's not you.
- This database error is caused by the Bike Forum servers being overwhelmed by "web crawlers", which "scrape data" from websites. It's like if every BF user, or many more than that, were all simultaneously flipping through every page on every thread, in order to read every post, all at once. In years past, the motive might have been to get data for targeted advertising and other things. Now, it seems to be powered by companies/entities/countries competing in Artificial Intelligence, which is now driven by "Large Language Models", which attempt to read every single thing published on the internet, and due to the feverish competition, do it frequently to build their database as quickly as possible. These data searches are supposed to be "polite", in a way that does not crash servers. Of late, they are not. The techs at BF have tried to detect this occurance, and put in a time delay or blocking a website, but it is difficult. I am guessing that web crawlers can easily change or disguise their identity and origin, making it difficult to permanently block a particular one.
- The BF servers had been down for many hours in the past, but in recent months, seems to reboot(?) in a matter of minutes when an error occurs. Not perfect, but way better than it was.
- Permanent resolution of this problem may require (only my guess), enforceable rules/legislation that would make these crawlers self-identify, follow the rules of the road, respect refusals, etc. That's way, WAY above my knowledge and understanding of the web, not my profession. If I can remember, next time I see someone I know who is retired from high up in computer tech, and is still active in trade associations, I'll ask if they know answers/suggestions on this.
Just a brief summary from early in this thread, for those new to it:
- This is not system-specific. Not your computer or phone, not operating system, not browser. It's not you.
- This database error is caused by the Bike Forum servers being overwhelmed by "web crawlers", which "scrape data" from websites. It's like if every BF user, or many more than that, were all simultaneously flipping through every page on every thread, in order to read every post, all at once. In years past, the motive might have been to get data for targeted advertising and other things. Now, it seems to be powered by companies/entities/countries competing in Artificial Intelligence, which is now driven by "Large Language Models", which attempt to read every single thing published on the internet, and due to the feverish competition, do it frequently to build their database as quickly as possible. These data searches are supposed to be "polite", in a way that does not crash servers. Of late, they are not. The techs at BF have tried to detect this occurance, and put in a time delay or blocking a website, but it is difficult. I am guessing that web crawlers can easily change or disguise their identity and origin, making it difficult to permanently block a particular one.
- The BF servers had been down for many hours in the past, but in recent months, seems to reboot(?) in a matter of minutes when an error occurs. Not perfect, but way better than it was.
- Permanent resolution of this problem may require (only my guess), enforceable rules/legislation that would make these crawlers self-identify, follow the rules of the road, respect refusals, etc. That's way, WAY above my knowledge and understanding of the web, not my profession. If I can remember, next time I see someone I know who is retired from high up in computer tech, and is still active in trade associations, I'll ask if they know answers/suggestions on this.
