SQL Clone
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


A Patch Disaster


A Patch Disaster

Author
Message
Steve Jones
Steve Jones
SSC Guru
SSC Guru (225K reputation)SSC Guru (225K reputation)SSC Guru (225K reputation)SSC Guru (225K reputation)SSC Guru (225K reputation)SSC Guru (225K reputation)SSC Guru (225K reputation)SSC Guru (225K reputation)

Group: Administrators
Points: 225090 Visits: 19638
Comments posted to this topic are about the item A Patch Disaster

Follow me on Twitter: @way0utwest
Forum Etiquette: How to post data/code on a forum to get the best help
My Blog: www.voiceofthedba.com
Scott.A.baron
Scott.A.baron
SSC Rookie
SSC Rookie (41 reputation)SSC Rookie (41 reputation)SSC Rookie (41 reputation)SSC Rookie (41 reputation)SSC Rookie (41 reputation)SSC Rookie (41 reputation)SSC Rookie (41 reputation)SSC Rookie (41 reputation)

Group: General Forum Members
Points: 41 Visits: 215
Interesting challenge as corporate infrastructure becomes more centralized and server virtualization expands. Fewer people support more platforms with automated tools. The day's of knowing or even identifying system support and admin teams seems like a distant memory.

Automated patch process in this environment is a challenge, all in the interest of cyber security and minimizing cost.
Chris Metzger
Chris Metzger
SSC-Addicted
SSC-Addicted (418 reputation)SSC-Addicted (418 reputation)SSC-Addicted (418 reputation)SSC-Addicted (418 reputation)SSC-Addicted (418 reputation)SSC-Addicted (418 reputation)SSC-Addicted (418 reputation)SSC-Addicted (418 reputation)

Group: General Forum Members
Points: 418 Visits: 89
Automation is actually easy and I have no idea how they managed this mistake unless they were using a 3rd party (ie. non-MS) tool to deploy the update. We use WSUS and don't EVER have this problem because WSUS works with Windows Update specifically to install only what's applicable to that OS (System Center does the same thing but is more complex and has more features) - and includes updates for all MS products (so I can apply SQL updates along with Windows updates and minimize downtime for everyone and then only have one reboot - all without having to pay it any attention). When done properly automated update maintenance works great - I know it saves me a lot of time and manual labor.
Nadrek
Nadrek
SSCertifiable
SSCertifiable (7.2K reputation)SSCertifiable (7.2K reputation)SSCertifiable (7.2K reputation)SSCertifiable (7.2K reputation)SSCertifiable (7.2K reputation)SSCertifiable (7.2K reputation)SSCertifiable (7.2K reputation)SSCertifiable (7.2K reputation)

Group: General Forum Members
Points: 7168 Visits: 2741
I'd have to say that we do see security patches for SQL Server these days, and they certainly require more work for me than service packs.
SP# > current SP, install it.

MS12-070 - well, what's your current build number? And if Microsoft Updates installed the one for the lesser build number, you can't upgrade to the greater build number (or at least I haven't figured out to go from build 10.00.5512 to 10.00.5826 without getting stopped by a "you've already installed this update" error)

Last year, SQL Server 2005 had two security updates (MS12-070, MS11-049) and no service packs.
2008 had one security update (MS12-070), no service packs.
2008R2 had one security update (MS12-070), one service pack
2012 had one security update (MS12-070) and one service pack.

Sum total: two service packs, five security updates.
Steve Jones
Steve Jones
SSC Guru
SSC Guru (225K reputation)SSC Guru (225K reputation)SSC Guru (225K reputation)SSC Guru (225K reputation)SSC Guru (225K reputation)SSC Guru (225K reputation)SSC Guru (225K reputation)SSC Guru (225K reputation)

Group: Administrators
Points: 225090 Visits: 19638
Nadrek (1/10/2013)

...
Sum total: two service packs, five security updates.


That's across 3 platforms, which still isnt' bad. If you look at the Oracle or DB2 lists, many more.

Follow me on Twitter: @way0utwest
Forum Etiquette: How to post data/code on a forum to get the best help
My Blog: www.voiceofthedba.com
Miles Neale
Miles Neale
SSCertifiable
SSCertifiable (5.3K reputation)SSCertifiable (5.3K reputation)SSCertifiable (5.3K reputation)SSCertifiable (5.3K reputation)SSCertifiable (5.3K reputation)SSCertifiable (5.3K reputation)SSCertifiable (5.3K reputation)SSCertifiable (5.3K reputation)

Group: General Forum Members
Points: 5262 Visits: 1695
If you consider how hard hackers and malware developers are trying to break in and corrupt or take data or software illegally this is rather surprising. And further if you consider the complexity and diversity of SQL Server functionality, having this few "fixes" is really amazing. I know the save them up and release a number of updates at one time, but still to not have one emergency security patch after another is great.

M.

Not all gray hairs are Dinosaurs!
lptech
lptech
Ten Centuries
Ten Centuries (1.3K reputation)Ten Centuries (1.3K reputation)Ten Centuries (1.3K reputation)Ten Centuries (1.3K reputation)Ten Centuries (1.3K reputation)Ten Centuries (1.3K reputation)Ten Centuries (1.3K reputation)Ten Centuries (1.3K reputation)

Group: General Forum Members
Points: 1273 Visits: 3506
From many rounds of patch management over the years, I would highly recommend the following:

1) Have a Computer Management Configuration Database (CMDB) in place, and make sure that the patch information is updated regularly thru an automated process.

2) If databases servers have their own CMDB, and there are good reasons to do so, make sure that the Windows support and patch management team(s) are aware of it. Provide them with the appropriate interface so they understand which are the DB servers. Not every junior Windows admin will realize that DB and web/app servers need to be treated differently.

3) SLA's for each server should be documented. It should be easy to group servers by application to see how the entire patching schedule should be set up. One of the more annoying things to deal with is spending Friday afternoon and evening re-schreschedulingxceptions to the company-wide patching window.

4) The patch coordinator for each application should be identified and easy to determine. Application inventory systems may only list the high level owner who won't always recognize 'ServerX'.

5) Make sure that you are getting the high availability from your clusters. In an active/passive setup, the passive node needs to be identified before every patching cycle. This should be automated, and fed into the process, as discussed in #1 & 2. Of course we would like every instance to be on the preferred node, but that doesn't always happen. Since cluster nodes are likely to have consecutive IP addresses and/or server names, both nodes could receive the patch at the same time, something that should obviously never happen.

The bottom line is that it is critical to have good processes in place. To quote the Yogi Berra Aflac commercial, when you don't have it, that's when you gotta have it.
TravisDBA
TravisDBA
SSCarpal Tunnel
SSCarpal Tunnel (4.5K reputation)SSCarpal Tunnel (4.5K reputation)SSCarpal Tunnel (4.5K reputation)SSCarpal Tunnel (4.5K reputation)SSCarpal Tunnel (4.5K reputation)SSCarpal Tunnel (4.5K reputation)SSCarpal Tunnel (4.5K reputation)SSCarpal Tunnel (4.5K reputation)

Group: General Forum Members
Points: 4524 Visits: 3069
Very good point Steve! Our security team just stopped the MS13-007 patch (http://support.microsoft.com/kb/2769327) here from going onto all of our servers when it was discovered through a website scan that applying that patch to our servers would bring down the websites. They scanned our websites and found hundreds of occurrences of the REPLACE function in the .aspx code. This just goes to show you that every patch that Mickeysoft puts out is not always in your best interest. You must examine each and every one of them on a case by case basis for your shops particular situation.:-D

"Technology is a weird thing. It brings you great gifts with one hand, and it stabs you in the back with the other. ...:-D"
Steve Jones
Steve Jones
SSC Guru
SSC Guru (225K reputation)SSC Guru (225K reputation)SSC Guru (225K reputation)SSC Guru (225K reputation)SSC Guru (225K reputation)SSC Guru (225K reputation)SSC Guru (225K reputation)SSC Guru (225K reputation)

Group: Administrators
Points: 225090 Visits: 19638
TravisDBA (1/10/2013)
Very good point Steve! Our security team just stopped the MS13-007 patch (http://support.microsoft.com/kb/2769327) here from going onto all of our servers when it was discovered through a website scan that applying that patch to our servers would bring down the websites. They scanned our websites and found hundreds of occurrences of the REPLACE function in the .aspx code. This just goes to show you that every patch that Mickeysoft puts out is not always in your best interest. You must examine each and every one of them on a case by case basis for your shops particular situation.:-D

Good catch.

There are definitely issues with some patches. I know I'm hesitant to apply any the first month. I'd rather let someone else test things.

Follow me on Twitter: @way0utwest
Forum Etiquette: How to post data/code on a forum to get the best help
My Blog: www.voiceofthedba.com
davoscollective
davoscollective
SSCrazy
SSCrazy (2.5K reputation)SSCrazy (2.5K reputation)SSCrazy (2.5K reputation)SSCrazy (2.5K reputation)SSCrazy (2.5K reputation)SSCrazy (2.5K reputation)SSCrazy (2.5K reputation)SSCrazy (2.5K reputation)

Group: General Forum Members
Points: 2535 Visits: 1008
Reading deeper into this story, it probably wasn't a patch to blame

"this was the result of Task Sequence distributed to a custom SCCM Collection. The Collection had been created/modified by an HP Engineer (adding a wildcard) and the engineer had inadvertently altered the Collection so that it was very similar in form and function to the “All Systems” Collection. The Task Sequence contained automation to format the disks"

That explains the no OS found error messages reported, and also the question of how MS OS patches could possibly be installed on the wrong systems.
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum

































































































































































SQLServerCentral


Search