this is my favorite memory

early 2006 we upgrade to SQL 2005. or early 2007. whenever SP2 came out. we spend a few months testing.

start the upgrade late friday or saturday night. we had a cluster running Win2000/SQL 2000 that we upgraded to SQL 2005. In place upgrade. everything goes OK and no complaints to the helpdesk the rest of the weekend.

Monday morning we get complaints of our main CRM app hanging. no idea why. check the console and we see the CPU spiking to 100% for up to 10 minutes at a time. our VP of IT thinks it's a virus. we open a case with PSS and they are swamped. that whole year you send them logs and don't expect to hear back for a week or so because they were so busy.

after a week of going back and forth with PSS and our own troubleshooting we find that it's a few problem queries causing the CPU spikes. one of them was selecting 20 or so columns with a where clause being up to a few hundred int values with an in(). nothing complicated and worked fine in SQL 2000. the SQL 2005 optimizer didn't like it even though it was an index seek or clustered index seek. forgot which one. Dev had to make changes to the query that night and the quick fix was a temp table.

Summary is that when we asked QA about it we found out that they would take the database that we restored and delete most of the data. a million row table would be trimmed to a hundred at most. just enough data for testing. and they would work from home a few days a week and what would happen they would click a button to test the app and go get coffee in the kitchen. come back a few minutes later and it would be done. no time testing to make sure that the action completed in a reasonable amount of time.