Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 

K. Brian Kelley - Databases, Infrastructure, and Security

IT Security, MySQL, Perl, SQL Server, and Windows technologies.

Rant: Applying Common Sense in IT

Common Sense...

I was watching a training video last night and the instructor made the statement we hear all the time:

"Test this in a non-production environment before you make a change in production."

In fact, he repeated it several times. In a recent article I wrote, I included the following warning:

 The nmap tool is often considered a hacking tool by organizations. If you're going to use this tool within your organization, make sure you have permission to do so by someone who is actually authorized to give you that permission. In alot of organizations, the DBA manager does not have this permission. It usually falls to a security or networking manager to authorize this. When you do get said permission, make sure it is in "writing." Email is fine. You don't want to go on verbal permission. If something goes wrong (you're in IT, so you always plan for something to go wrong, or at least you should be) you want to be able to show you had permission to do what you were doing. Failure to get written permission from the right person can lead to what we call a "career altering event." Avoid this by doing the legwork to get the permission.

Then I stopped and asked the question, "Why do we have to issue these warnings?" After all, we're in IT. We work with systems that apply a strict set of operational logic in a particular order based on what we tell them. We often lament about end users who do very illogical things. So why do we continually have to provide others with warnings about testing something before rolling it into production or about getting permission to run a tool that could be seen as a hacker tool?

The short answer is because we probably aren't thinking as much as we should be. Before we do something we're likely not asking ourselves the implications of what we're about to do. Or we're so rushed to get something in that we don't have time to give it some serious thought. Computers allow us to work faster, but we shouldn't let that cause us to work too fast. We need to slow down. We need to understand what we're doing and why we're doing it. But we don't. And when we don't is when we make a mistake we could have avoided. We bring down a server we shouldn't have. We delete a database because we weren't paying adequate attention.

We also need to insist upon common sense and logic be applied by our IT co-workers. One does not roll something into production without testing first. One does not go on the first article or blog post hit in Google/Bing. Things like that. I think if we did, end users would have more confidence in us. And we'd get more done.

Comments

Posted by Steve Jones - SSC Editor on 18 August 2012

I'm not sure what the problem is here. I think some of it is that too many of us learn about certain situations, certain reactions, and we don't often have the vision to understand potential implications.

Running nmap? If you've never run it, and it's a scanner, it might be easy to think this is non-intrusive and won't cause an issue.

I know it sounds like common sense, and perhaps it is, but sometimes I wonder about the way us experienced folks view the world. Too many people today have much narrower experiences.

The testing in non-production is very common sense, but too many people don't follow good deployment practices anyway, or they don't know how to test well, and their execution in production is less than ideal.

Posted by ramaratna on 22 August 2012

having common sense is part if good development skills. which is sinking

Posted by raotor on 23 August 2012

Personally, I think much of the lack of common sense in IT comes about due to a lack of discipline and not taking a slower, more measured and methodical approach. Too many organizations want everything done yesterday and many managers pass this pressure down to the folks on the ground who need to actually do the work without fully appreciating or setting realistic time frames to their superiors for the work.

In my experience, it's always better to take a little more time to do something and get it right then rushing things through and getting it wrong.

Also, personally speaking, having come from a 3GL background, I think that instils more discipline about the approach you take to development and in particular testing. I do wonder if sometimes the point and click culture we have today requires less rigour and promotes the whole notion of getting things done faster - which is not always better.

Of course, there are many other reasons too.

Posted by paul.knibbs on 23 August 2012

I don't think it's a matter of common sense here--it's usually a matter of we're all busy and doing stuff through the proper channels takes a long time. Maybe people will short-cut in order to avoid that extra time? You're right, it might well cause a big issue if they make a mistake, but with job pressures as they are, I certainly wouldn't blame anyone for doing it.

Posted by hfxDBA on 23 August 2012

... as the old saying goes... the thing about common sense is, it's not really that common.  Your common sense sounds a lot like mine, but that doesn't mean it matches what others' common sense is.  Even if you have common sense (like yours), most people are lazy by nature and don't care about common sense.  That is my cynical view of my colleagues :)  I often get the rolling eyes when I insist on certain practices as i am sure you do as well.

Posted by Dennis Wagner-347763 on 23 August 2012

It never hurts to "overengineer" a procedure, so restating a warning message is definitely needed.  At least the person who didn't heed the warning was aware of the dangers.

A lot of times is it is upper management who causes us to work too quickly with unrealistic deadlines or even worse, poor planning that becomes "our crisis."  I work in an industry that builds hotels, and I always have to use that as an analogy when dealing with upper management.  "You wouldn't want your contractor to change the construction drawing on the new two-story, 100 room hotel to a ten-story, 500 room hotel by making changes on the plans in one hour!  People would get hurt physically when it collapses because the proper planning was not used to determine load balances, structure stability, etc."  

The same is true in IT.  When we are told to build something in a week that should have taken two or three just to plan out the specs, things WILL go wrong.  It won't be upper management's responsibility to fix the issues for the next six months, it will be ours, in addition to the other 500 things you want us to do.  I suppose that is why there are so many highly qualified IT people looking for jobs these days.  They either built things that didn't work properly or they stood up to upper management and didn't get things done quickly enough.

Posted by roger.plowman on 23 August 2012

Never time to do it right, always time to do it over.

Posted by Ken Mercadante on 23 August 2012

"Common Sense" is simply a shorthand for collective experience. It's only common because it happens to most people, and most people learn the same lessons from that experience. It's a lot like statistics: it works well when applied in the aggregate, but at the individual level it's no more predictive than a coin toss.

Unfortunately our world has accellerated since this term was coined. People come into IT through many paths (and databases have even more points of entry) and as the years go by we can assume less and less about the collective experience anyone brings to the job.

And even if it is commonly known, reminders never hurt.

Posted by emoore 99634 on 23 August 2012

With technology churn like it where it appears anything you learn is obsolete within 18-24 months, how deep can your pool of expertise be...  What age and wisdom tells you is just don't jump into that pool before ascertaining how deep it is.

Posted by johnpatrick.garza on 23 August 2012

"Why do we have to issue these warnings?"  Because it's common sense to do so....  

Posted by jbarnes 79019 on 23 August 2012

Thank God for dumb people!  They ensure that I will always have a job.

Posted by dan-572483 on 23 August 2012

There are often situations where we're 99 percent certain something is not going to cause an issue.  It's for the other 1 percent that we test everything on nonProduction first, and get permission in writing for anything that has a 1 percent uncertainty.

Posted by SAinCA on 23 August 2012

Sometimes there's no alternative BUT to test in production...  Why?  Consider my case that happened only two weeks ago: when data aren't flowing in because someone changed a configuration at the client site and there's no way to change it back (PC died and client insisted on renaming the new PC!), and no way to test any kind of EFIX in any environment without losing some of the Client's data (unacceptable), one has to think outside the box!

Rule #1: "First do no harm", so always bound the test script with a BEGIN TRAN ... ROLLBACK.

Rule #2: NEVER change the production code itself.  Make copies of the affected SP's/UDFs and change those.

Rule #3: Trap the incoming failed requests with Profiler and use them against the copied artifacts, inside the BEGIN TRAN/ROLLBACK envelope.

Rule #4: Don't just test ONE incoming case unless that's all you have.  Grab as many as makes sense to test conclusively.

Rule #5: If you're not a one-man-band get someone else's eyes on it!

Rule #6: Take some time to REALLY consider any other ramifications.  (In my case, a couple of Alert emails wouldn't work for that site, so a couple of additional SPs were cloned and tested using the do-no-harm paradigm.)

Rule #7: Be patient!  Managers and developers alike.  5 more minutes to do it right vs. hours to fix an "I didn't see that coming" is priceless.

Totally agree with the "apply common sense" reminder, BTW.  Do keep the reminders coming...

Posted by mike.broniszewski on 23 August 2012

I think 'common sense' is only a small part of the issue. In my experiance I see people use their common sense every day when making similar decisions. What I also see is that when they are looking at tools or looking at the steps they need to follow is that most people do not plan things out with the proper people involved. For example, you can test a data fix script in a test environment and get all neccessary approvals to run it in production but if you haven't talked to your data warehouse team you are probably going to cause issue. Proper communication is neccessary. To expand on SAinCA's list...

Rule #8: ensure all impacted parties know what is happening before it happens.

Posted by mike.broniszewski on 23 August 2012

I think 'common sense' is only a small part of the issue. In my experiance I see people use their common sense every day when making similar decisions. What I also see is that when they are looking at tools or looking at the steps they need to follow is that most people do not plan things out with the proper people involved. For example, you can test a data fix script in a test environment and get all neccessary approvals to run it in production but if you haven't talked to your data warehouse team you are probably going to cause issue. Proper communication is neccessary. To expand on SAinCA's list...

Rule #8: ensure all impacted parties know what is happening before it happens.

Posted by T_A_D on 23 August 2012

I am one of those presenters that give those annoying admonitions. Whenever I give training about operations that have potential for failure, I remind people to be careful and test on a development system. Training videos are often viewed by less experienced people who can benefit from the reminders. A warning is no substitute for painful real-life lessons, but you must admit that they are somewhat easier to tolerate.  Also, reminders may help the more experienced pause for one important moment and think about implications.

Posted by dan-572483 on 23 August 2012

There are often situations where we're 99 percent certain something is not going to cause an issue.  It's for the other 1 percent that we test everything on nonProduction first, and get permission in writing for anything that has a 1 percent uncertainty.

Leave a Comment

Please register or log in to leave a comment.