SQL Clone
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


Deployment Failures


Deployment Failures

Author
Message
Steve Jones
Steve Jones
SSC Guru
SSC Guru (225K reputation)SSC Guru (225K reputation)SSC Guru (225K reputation)SSC Guru (225K reputation)SSC Guru (225K reputation)SSC Guru (225K reputation)SSC Guru (225K reputation)SSC Guru (225K reputation)

Group: Administrators
Points: 225190 Visits: 19641
Comments posted to this topic are about the item Deployment Failures

Follow me on Twitter: @way0utwest
Forum Etiquette: How to post data/code on a forum to get the best help
My Blog: www.voiceofthedba.com
Graeme100
Graeme100
Hall of Fame
Hall of Fame (3.4K reputation)Hall of Fame (3.4K reputation)Hall of Fame (3.4K reputation)Hall of Fame (3.4K reputation)Hall of Fame (3.4K reputation)Hall of Fame (3.4K reputation)Hall of Fame (3.4K reputation)Hall of Fame (3.4K reputation)

Group: General Forum Members
Points: 3368 Visits: 795
Yes indeed. A rather touchy subject for me Smile

Our deployments used to be handled by DBAs. We would do all the database changes in a way we could manage and we bcode was pretty easy to roll out via a copy paste type scenario. Basic but whoever was doing it knew what they were were doing.

Recent developments in web code design plus the way we manage database deployments (namely via TFS and powershell scripts) means that the DBAs, that now have to deploy, know a lot less about the deployment process and things like the DB changes are wrapped up in a TFS deployment package. Whilst I see the benefits of this, it MUST work correctly and I guess our process is not working right. We find a simple deployment of say a few changes requires a complete schema compare and removal of any adhoc processes such as replication. I think this could work better but our hands are now tied by the development dept.

In my experience I find that these newer "black box" type deployments leave me and who ever does them feeling rather sceptical and indeed nervous.

What these newer techniques have resulted in are the requirement of a deployment team rather than one or two individuals. Not sure if that is good/bad, efficient/inefficient. All I know is that now we approach deployments with far more fingers crossed than I feel is necessary

Graeme



Bob JH Cullen
Bob JH Cullen
Ten Centuries
Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)

Group: General Forum Members
Points: 1248 Visits: 786
We run a small (5 server) computer telephony platform that has to run 24/7. Finding any maintenance window is a major problem - we have several hundred client services, all interacting with real-time databases and the risk of something failing to deploy is too horrid to contemplate.
There is no budget for a replicated test environment, so we have one platform that is dev/test/staging/production. Yeah, yeah, I know ...
So when we deployed a new co-lo platform a year or two back, we took the decision (rightly or wrongly) to get everything up to date prior to go-live and then freeze the build on all servers. The only thing I've done since has been a couple of critical patches on the blade enclosure and SAN controllers when we needed to shut the whole system down for two hours due to site power issues - and that exercise took three weeks to achieve due to all the client consultations and notifications that were needed.
Much clenching, I can tell you, when the SAN didn't restart first time !
So, I'm happy to be accused of ostrich-like behaviour, but I'd rather avoid patches, SPs and updates. If it ain't broke, etc. etc.
SanDroid
SanDroid
Hall of Fame
Hall of Fame (3.4K reputation)Hall of Fame (3.4K reputation)Hall of Fame (3.4K reputation)Hall of Fame (3.4K reputation)Hall of Fame (3.4K reputation)Hall of Fame (3.4K reputation)Hall of Fame (3.4K reputation)Hall of Fame (3.4K reputation)

Group: General Forum Members
Points: 3362 Visits: 1046
we often fine broken features limping along for months or years

O_o If only this were true. Cool
Eric M Russell
Eric M Russell
SSC-Forever
SSC-Forever (43K reputation)SSC-Forever (43K reputation)SSC-Forever (43K reputation)SSC-Forever (43K reputation)SSC-Forever (43K reputation)SSC-Forever (43K reputation)SSC-Forever (43K reputation)SSC-Forever (43K reputation)

Group: General Forum Members
Points: 43621 Visits: 12057
Database deployments have unique challenges like: altering or dropping/recreating schemas while preserving existing data, object dependencies, schema variations between development, QA, UAT, and production.
However, the most problematic issue is how to rollback to a previous pre-deployment state. Sure, you can wrap your schema alterations and row insert/update/deletes in a transaction and then ROLLBACK cleanly during deployment in the event of a runtime error. But what if the business requests a rollback the following day, hours after the users have been using the application? That's why the best approach in many situations is to analyze what specifically went wrong, and then deploy a followup fix.


"The universe is complicated and for the most part beyond your control, but your life is only as complicated as you choose it to be."
dduensing
dduensing
SSC-Enthusiastic
SSC-Enthusiastic (160 reputation)SSC-Enthusiastic (160 reputation)SSC-Enthusiastic (160 reputation)SSC-Enthusiastic (160 reputation)SSC-Enthusiastic (160 reputation)SSC-Enthusiastic (160 reputation)SSC-Enthusiastic (160 reputation)SSC-Enthusiastic (160 reputation)

Group: General Forum Members
Points: 160 Visits: 162
As a fan of the "consistent environment" I am a firm believer in the value of a constant, scheduled update process, available via scheduled, and ad hoc delivery. We have had a few "issues" related to some updates, primarilly in older software, that is kept in production because of a business process, to "delicate" to bring up to current levels. In the relatively lengthy periiod of time I've been maintaining systems,

I think I have seen less than five significant issues, caused by a vendor supplied update. Our AV vendore delivered an update which not only broke AV, but made it impossible to repair without a local machine touch... (Tight looped to the point of useless...) ODBC updates were problematic for a period of time, but that was generally a mashup of different vendors supplying the same files... Internally I have seen issues with in house software, mostly related to a "siloed" test platform, where the "it worked on my machine" reasoning is applied. The Dev->Test->Stage->Prod model is obviously the safest, although in some ways brings its own challenges. Duplicating every system is obviously not a real option.

Our model is still update Test/Dev first, watch for fallout, and continue if no noise. The schedule helps us to anticipate issues, people may blame the updates for unexpected behaviour, but generally the update is not the root of the problem.



SanDroid
SanDroid
Hall of Fame
Hall of Fame (3.4K reputation)Hall of Fame (3.4K reputation)Hall of Fame (3.4K reputation)Hall of Fame (3.4K reputation)Hall of Fame (3.4K reputation)Hall of Fame (3.4K reputation)Hall of Fame (3.4K reputation)Hall of Fame (3.4K reputation)

Group: General Forum Members
Points: 3362 Visits: 1046
Eric M Russell (12/14/2012)
Database deployments have unique challenges like: altering or dropping/recreating schemas while preserving existing data, object dependencies, schema variations between development, QA, UAT, and production.
However, the most problematic issue is how to rollback to a previous pre-deployment state. Sure, you can wrap your schema alterations and row insert/update/deletes in a transaction and then ROLLBACK cleanly during deployment in the event of a runtime error. But what if the business requests a rollback the following day, hours after the users have been using the application? That's why the best approach in many situations is to analyze what specifically went wrong, and then deploy a followup fix.

+1000 agree with this. I have lived this many times. You deploy a release and find a use case nobody tested or was aware of that is causing issues in another section of code. Is it better to roll back to the previous version that can no longer work with the bussiness data, or fix the two lines of code and release a patch?
I vote for the patch every time. :-D
Miles Neale
Miles Neale
SSCertifiable
SSCertifiable (5.3K reputation)SSCertifiable (5.3K reputation)SSCertifiable (5.3K reputation)SSCertifiable (5.3K reputation)SSCertifiable (5.3K reputation)SSCertifiable (5.3K reputation)SSCertifiable (5.3K reputation)SSCertifiable (5.3K reputation)

Group: General Forum Members
Points: 5264 Visits: 1695
We have a regular schedule for both system and application updates. If something fails it is rolled back. We have also failed forward on one or two things but prefer not to do that. It often leads to a bigger failure down the road.

From time to time a SP from Microsoft might be missed but once noticed it is patched in the next available window.

Better safe then down!

M.

Not all gray hairs are Dinosaurs!
ken.trock
ken.trock
SSCommitted
SSCommitted (2K reputation)SSCommitted (2K reputation)SSCommitted (2K reputation)SSCommitted (2K reputation)SSCommitted (2K reputation)SSCommitted (2K reputation)SSCommitted (2K reputation)SSCommitted (2K reputation)

Group: General Forum Members
Points: 1963 Visits: 1730
Years ago the company I worked for would patch the majority of our servers one Friday night each month. The Microsoft patches for the month, and other software patches, would be bundled up into SMS (Systems Management Server) packages and deployed to thousands of servers.


That's pretty much how our server support teams do it in our data centers and they're pretty good at. But this is applying up to a dozen patches on a couple thousand machines. Some database (SQL Server, maybe Oracle and others), Exchange, web, application. The chance of something going wrong on at least some of these is high. On the SQL Server side I've seen issues where we've lost drivers, had authentication mode switched behind our backs, and other problems that required another reboot.

I'm not saying this isn't the way to go, but a lot of checking has to be done after each cycle to be sure.

Ken
Dave Poole
Dave Poole
SSC-Insane
SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)

Group: General Forum Members
Points: 24307 Visits: 3494
For us it varies. The biggest challenges come when new technology is adopted.
We know what it takes to make a successful and uneventful SQL Server deployment but by definition new technology is...well.....new.

The technology estate gets more complicated every year. What is extremely important is that some rigour is put in place with regard to deprecating old stuff. You can't just keep adding and adding new stuff and expect the same number of staff to keep the stuff working. Unfortunately deprecating stuff is not a revenue generating activity so some other lever is needed to get this brought to the fore.

Revenue generation is much sexier than cost control.

LinkedIn Profile
www.simple-talk.com
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum

































































































































































SQLServerCentral


Search