Worst ever day for this. Site showed down on downdetector. Off for hours on end, I think over 5 hours early this morning. Even once back up, frequently went down again. Curious if this is due to external site attacks, overwhelmed with spam posts, or internal software or hardware issues or external web support? *No complaints*, I'm sure the staff are pedaling as hard as they can, just wondering. EDIT: Ah, I see brief tech note above about this. Good wishes your way. (Looks up web crawler...)
From wiki, bold emphasis mine:
Web
search engines and some other
websites use Web crawling or spidering
software to update their
web content or indices of other sites' web content. Web crawlers copy pages for processing by a search engine, which
indexes the downloaded pages so that users can search more efficiently.
Crawlers consume resources on visited systems and often visit sites unprompted. Issues of schedule, load, and "politeness" come into play when large collections of pages are accessed. Mechanisms exist for public sites not wishing to be crawled to make this known to the crawling agent. For example, including a
file can request
bots to index only parts of a website, or nothing at all.
From search earlier today of <bikeforums database error> that showed beginning, but would not fully display due to website down; Now can be read in full. This from post in 2007, bold emphasis mine:
Just a quick technical update.
The database errors and overall slowness we experienced was cause by a highly aggressive (probably malicious) spider that was not interested in maintaining a courteous frequency of visits to BikeForums. This caused one of our three web servers to loose connection to the database and resulted in displaying db errors.
We have made some tweaks to our server monitoring which will enable us to resolve the issue much quicker in the future. Thank you for expressing your concern and your patience while we worked the issue out.