One of the nastiest surprises I ever had as a developer was to be confronted with a specialized team in the corporate IT Test department, who called themselves the ‘Limit Testers’. We were just easing an application out of the door into production when they turned up, all smiles. They were sharpening their pencils to start what is now generally called Recovery Testing. Their task was to test whether the application recovered from unpleasant events such as shortage of disk space, unexpected loss of communication, or power out conditions, without loss of data or functionality.
Any Developers will be as likely to countenance this as drop their own babies to see if they bounced. On the contrary these good folks, laughed heartily as they proceeded to destroy our pre-production test cell. Vainly, I protested that power brown-outs wouldn’t happen to our production systems. My spluttered protests, when they tugged away at the spaghetti connection behind the racks, went unheeded. They ruefully recounted a series of painful stories of unpleasant events that had happened in our own expensive production environment. Human error or malice can confound the cleverest hardware-based resilience mechanism.
After the horror came the learning. Business systems must be designed to intrinsically support the integrity of business processes. Business data must never be potentially volatile unless the business processes are atomic, consistent, isolated and durable. The cool distributed business layer we’d constructed was useless without a resilient messaging system that could manage groups of transactions that represented business processes, or ‘Sagas’. However, my Limit testers couldn’t break the integrity of the database data even when they killed the server.
The experience made me permanently suspicious of the siren calls that suggest that it is easier, and more appropriate, to place business logic outside the database. We sometimes forget the essential benefits that a full-specification relational database brings. SQL Server would be lightweight, faster and cheaper were it not for the resilience that it offers to transactions; Oh yes, it is faster and easier for developers to put business logic into middle-layers, up to that point where the Recovery Testers visit.
