It was kind of amazing to see SQLServerCentral have issues over the last few days. We’re still not sure what happened, but the load went crazy over the last few days, substantially slowing email sends and causing performance problems.

There are definitely some database, issues, and we are looking to provision a larger database server in the short term. That’s not a great fix, but we’re a business like any other and hardware is often quicker and easier to deal with than code in the very short term. As we dug in, we realized the same database server we’ve had for over 2 years, with more than double the load, just can’t keep up. I’m not sure if anything else has changed code-wise recently, but the quick fix is a new server. Since we have a hosted server with Rackspace, this is actually a fairly easy and relatively inexpensive fix.

Beyond that, we have some code to examine. As with most businesses, we had developers built the site, without a real DBA working alongside them. It wasn’t a great solution, but it also is the realization of how resources sometimes get deployed. I think our developers did some neat things in one sense, not so good in another. There is a boatload of nHibernate code in there, and some of those pages appear to be amazing POS constructs. If nothing else, I think I would be terrified of ever using nHibernate in any project just based on my experiences here.

As things seemed to fail today, we made the decision to move the forum database to our backup server. We’ve kept a second one handy for DR purposes, and since the code is fairly separate, it seemed like an easy fix to reduce the load.

Brad McGehee expressed concerns, which seem to be well founded. I’m having issues working in the forums, so I think there are still things broken.

In one sense this is the opposite problem that we had for years. Early on, Andy, Brian, and I coded quite a few things on the site, often in a half-a**ed way, since we weren’t programmers. We did have lots of RI and normalized data, but the front end code wasn’t anything to showcase. Now it seems we’re in the reverse situation. There is a lot of nice front end code, well documented and structured, but the database is a bit of a mess.

Where we go from here I’m not sure, but I expect that the next couple of weeks will involve some root cause analysis as well as some refactoring of code. I’d prefer to throw out the nHibernate design and build a simpler structure that is easier to tune, as well as maintain, but I’m not sure that will happen.

However I will try to continue to update you with information about how we will proceed.