Worst Practices - Making a "Live" Change

SQLServerCentral is supported by Red Gate Software Ltd.

Popular Topics

Home

Members

Calendar

Who's On

Home » Article Discussions » Article Discussions by Author » Discuss Content Posted by Steve Jones » Worst Practices - Making a "Live" Change

21 posts, Page 1 of 3

1 2 3 »»»

Worst Practices - Making a "Live" Change

Rate Topic

Display Mode

Topic Options

Author

Message

Steve Jones - SSC Editor

Posted Tuesday, December 24, 2002 12:00 AM

SSC-Dedicated

Group: Administrators
Last Login: Yesterday @ 3:19 PM
Points: 31,526, Visits: 13,863

Comments posted to this topic are about the content posted at http://www.sqlservercentral.com/columnists/sjones/worstpracticesmakingalivechange.asp

Follow me on Twitter: @way0utwest

Forum Etiquette: How to post data/code on a forum to get the best help

Post #8983

ckempste

Posted Monday, January 20, 2003 1:41 AM

SSC Eights!

Group: General Forum Members
Last Login: Tuesday, July 31, 2007 8:20 AM
Points: 885, Visits: 1

Hi Steve

As an apps dba, you wouldnt believe the excuses and comments from developers re getting so called adhoc changes through test then prod with a time span of 1 minute. The development manager really needs to own and manage suck practices and be thoroughly supportive of proper change practices and impact analysis. People tend to forget how much it can cost companies for those "minor patches" going wrong.

Cheers

Chris K

Chris Kempster
www.chriskempster.com
Author of "SQL Server Backup, Recovery & Troubleshooting"
Author of "SQL Server 2k for the Oracle DBA"

Post #50323

Antares686

Posted Monday, January 20, 2003 6:07 AM

SSCrazy Eights

Group: Moderators
Last Login: Yesterday @ 10:09 AM
Points: 8,357, Visits: 685

Generally it is the person who normally barks about these things that is the one who fouls things up in groups I have worked with. However, 90% of the time they make changes right before going on vacation so everyone else has to determine the mistake in their code. And the worst part of all is they tested the code in the test environment but instead of scripting the object or code changes they type them by hand.

Post #50324

David.Poole

Posted Monday, January 20, 2003 10:35 AM

SSCrazy

Group: General Forum Members
Last Login: Yesterday @ 1:21 AM
Points: 2,764, Visits: 1,439

Gods do I hear you on this, and I'm deaf!!

This is why I always script everything I do. Yes EM is quick and easy, but the wretched thing should be locked down in the live environment!

We follow the following procedures

Unit test our code
Script and document the installation procedure
Get someone else to work throught the script, a curmudgeonly BOFH is best for this.
Get someone else to test the results, the user from hell is best for this

If neither the BOFH or user smiles at you then your change and installation procedure works and can be applied to the live system.

You would think that this would be fool-proof, but we often deal with 3rd party suppliers. Unless all suppliers buy into this test mechanism then you may still find that even the most carefully planned change can still go wrong.

LinkedIn Profile

Post #50325

abhi_develops

Posted Monday, January 20, 2003 9:14 PM

SSC Rookie

Group: General Forum Members
Last Login: Thursday, October 22, 2009 11:55 AM
Points: 34, Visits: 2

Our product has been implemented across several sites. We use SQL Server.
What has come to our notice is that too much time is being spent fixing the sameproblems over and over again. This is due to the fact that some one at a particular site notices a problem, fixes it up and doesn't bother to inform others about it. When the same problem crops up at another site, the person over there fixes it up as well.
The problem, we now have two different version of codes since PERSON A failed to inform the central team about the problem encountered since he or she thought it was not 'worth informing'. We have now started a code consolidation exercise which is going on forever.

This is another worst practice. Not bothering to inform 'trivial changes'.

Cheers!
Abhijit

Post #50326

RyanRandall

Posted Tuesday, January 21, 2003 4:27 AM

SSCommitted

Group: General Forum Members
Last Login: Tuesday, May 29, 2012 11:22 AM
Points: 1,755, Visits: 4,652

I totally agree with the article, and all subsequent comments.

I'd like some debate, however, so I'd like to state 2 extremes, and ask where people's tradeoffs are?

1. Critical system, 24 hour access, any downtime at all is damaging.

2. Non-critical system, used every Friday for 5 minutes, doesn't matter if it doesn't work for a month or so.

I'd be surprised if there is anyone who would make changes to a live 1, and I'd also be surprised if there is anyone who would NOT make changes to a live 2.

There must therefore be a middle-ground somewhere. Ultimately this middle-ground should be decided based on cost and business risk, but I would be interested to know where YOUR middle-ground is...

Ryan Randall

Solutions are easy. Understanding the problem, now, that's the hard part.

Post #50327

David.Poole

Posted Tuesday, January 21, 2003 4:51 AM

SSCrazy

Group: General Forum Members
Last Login: Yesterday @ 1:21 AM
Points: 2,764, Visits: 1,439

My experience with systems that are used periodically i.e. at set intervals but not continuously, is that when they are used it is "all hands to the pump" and they absolutely have to be up and running when they are needed.

There is a danger that an important repair is forgotten and you are left desperately trying to bring yourself back up to speed when it becomes an urgent issue.

The time to be pulling your hair out with these things is not when their failure is highly visible and everyone is looking at you.

I think the danger of having a "it doesn't matter if this doesn't work for a while" application is that it encourages sloppy practices and poor discipline.

It is very easy to fall into bad habits and to be lulled into complacency. It is only when you are badly bitten in the bum that you realise jsut how badly you lulled.

LinkedIn Profile

Post #50328

Antares686

Posted Tuesday, January 21, 2003 5:10 AM

SSCrazy Eights

Group: Moderators
Last Login: Yesterday @ 10:09 AM
Points: 8,357, Visits: 685

No middle ground for me sorry. Even thou item 2 is non-critical and can stand to be down for a month you will still have to fend off customer complaints concenring it being down. And many will rip you a new one if it goes down more than once (I know I have had this happen to a developer here who learned his lesson). Good coding and implementation practices should always be used and tested no matter the scenario. Besides no matter how non-critical a system is, any amount of downtime and troubleshooting is a waste of time, energy and resources. Ultimately even thou it may be insignificant it will cost you in dollar amounts and the more mistakes made the higher the amount.

Post #50329

RyanRandall

Posted Tuesday, January 21, 2003 11:46 AM

SSCommitted

Group: General Forum Members
Last Login: Tuesday, May 29, 2012 11:22 AM
Points: 1,755, Visits: 4,652

I'm a bit confused by those replies.

David - If the system has to be up and running when it's needed, then is that a non-critical system?

Antares:
a) Scenario 2 isn't necessarily a customer system?
b) I don't follow your point about downtime and troubleshooting being a waste of time, energy and resources in the context of making some changes on either the live or the development version.

By the way, how do you guys copy your changes from the development to the live system? Do you have a system in place to do it? If so, how do you manage changes to that system?

I should say that I'm not advocating any approach, I'm just asking a few questions - hopefully to try to get a discussion going.

Ryan Randall

Solutions are easy. Understanding the problem, now, that's the hard part.

Post #50330

Antares686

Posted Tuesday, January 21, 2003 11:52 AM

SSCrazy Eights

Group: Moderators
Last Login: Yesterday @ 10:09 AM
Points: 8,357, Visits: 685

quote:

I'm a bit confused by those replies.

David - If the system has to be up and running when it's needed, then is that a non-critical system?

Antares:
a) Scenario 2 isn't necessarily a customer system?
b) I don't follow your point about downtime and troubleshooting being a waste of time, energy and resources in the context of making some changes on either the live or the development version.

Sorry should have stated specifically it is a waste in production. All should be tested prior ro production. If tested properly then no issue should arise (except the occasional unknown factor). The key is production should only be declared when all sanity checks have been done otherwise it is better to call it a BETA, meaning bugs may exist not neccessarily do exist.

And as for not being a customer system then it doesn't exist. If you build a system and store data to be used by anyone it is a customer system. Even if it is just one person. Otherwise you are talking Dev system or Test system. I am referring to Prod which means it is in use.

Edited by - antares686 on 01/21/2003 11:55:08 AM

Post #50331

« Prev Topic | Next Topic »

21 posts, Page 1 of 3

1 2 3 »»»

Permissions