Feedback Recent Downtime

Kevin

Kevin

Code Monkey
Staff member
Well, everything has been back up online for about 36 hours now with no noticeable problems.

Essentially I tasked myself with answering a single question: On our dedicated server, which should be more than sufficient, we have several vBulletin installations of various sizes but why is it only Cool Sci-Fi that is experiencing the MySQL timeouts?

I ended up going through just about every vBulletin option available as well going through the code of several add-ons being used to find out what's going on. In the end I came across several chunks of code that would've added extra load on the SQL tables in additional to some schedule tasks that were constantly updating counters. I also come across some situations with the Usenet threads in the alt.test group that had binary attachments.

In short... I believe a combination of factors were at play and I've fixed most of the ones that were within reach.

Next up would be to work on some of more complicated issues as well as slowly turning back on some of the various options that were disabled.

EDIT: I'm also on the verge of deciding to not re-enabling vBSEO here at Cool Sci-Fi. While it definitely has some advantages & nifty features I'm finding it to be the biggest 'mystery box' when it comes time to researching problems.
 
One decision down... I have decided to leave vBSEO turned off. At the end of the day it came down to deciding whether its benefits outweighs the slight extra load it puts on the server. I liked the benefits of it, I really do, but Cool Sci-Fi is the largest & busiest site on our server so any extra burden means a degradation in performance.
 
One author was promoting the site a few days ago on his blog. Not good timing during the downtime obviously, but it happens.

Was vBSEO seen to be the problem then for definite? Didn't like some formatting of posts then? COnflicting with binaries, etc.
 
Tim said:
One author was promoting the site a few days ago on his blog. Not good timing during the downtime obviously, but it happens.

Was vBSEO seen to be the problem then for definite? Didn't like some formatting of posts then? COnflicting with binaries, etc.
Click to expand...
The 'ultimate' problem, I believe, was a "Statistics" module that used to be on the front page in the lower left column. It would display the total number of threads & posts on the site along with the total number of members and the name of the newest member. The stats were being updated in real time so every time there was thread or post made the counters were being updated. Combine that with something like the Usenet feeds where dozens of new threads could be created in a single minute and the system was just getting hammered. The scheduling of the updating of the statistics could be scheduled to always update or on the hour (eg: every X hours). For now I've turned it off entirely because even if I had it update once an hour or once every two or three hours then it would've meant that there still would've been problems at some point during the day temporarily while the update was occurring.

Compounding the problem was the Usenet group named alt.test that is used for testing. It is supposed to be a text only newsgroup but people were posting binary files in there. In Usenet groups, of course, binary files are broken down as MIME text and are display as any other message. That means when looking at the thread it would look like a huge thread with nothing but junk in it but to a Usenet client (eg: Agent or Outlook Express or others) it would be translated from MIME back into a binary attachment. When those threads were being imported into vBulletin I believe what was happening is that it was trying to index all of that MIME 'binary' text in the search tables causing some huge overhead in the MySQL table indexes. It was the deletion of those threads that was actually the cause of the site being down for a few days since deleting even a single thread was taking minutes! :eek:

Then to a lesser degree were some items such as vBSEO where it had to do some small queries to turn the default URLs into the English URLs. I was also taking advantage of some other settings that the software offers. In the end I decided to take the plunge and not re-enabling it since every time there is a performance issue with the site it is one of the few packages that is a bit of a 'black box' in regards to being able to take a look at the actual code that is being executed and what it is doing since the code is obfuscated.

Also on the list where some vBulletin plugins that were executing queries that weren't needed. One example is another module for the front page for member information. The 'Welcome' module is essentially a duplicate of the user information in the menu bat at the top of the screen in that it allows the user to log in/out and also shows them the number of new messages since their last visit and how many private messages they have. Because it is redundant having it on the screen twice I have the module disabled. Even though the module is disabled the plugins were still being executed so that every time a new thread was created a query was executing to update all of the users records to increase the number of new posts since their last visit.

I also tweaked a few other items to cut down on the number of queries being executed but the items above are the big ones.
 
www.realitymod.com using Linkbacks plugin. The forum has been slow, leading to double posts and internal database errors, usually with new thread posting or trying to edit a post, specifically in Advanced mode

They just updated their vB software over the weekend and still having the problem. They thought they knew what they were doing, it appears the plugin writers don't have everything sewn up!
 
A few weeks later and all seems to be stable yet. :cool: There was a weird quirk yesterday but nothing else that I'm aware of.
 
Back
Top