Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase 12»»

A Patch Disaster Expand / Collapse
Author
Message
Posted Thursday, January 10, 2013 12:14 AM


SSC-Dedicated

SSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-Dedicated

Group: Administrators
Last Login: Today @ 2:04 PM
Points: 32,780, Visits: 14,939
Comments posted to this topic are about the item A Patch Disaster






Follow me on Twitter: @way0utwest

Forum Etiquette: How to post data/code on a forum to get the best help
Post #1405231
Posted Thursday, January 10, 2013 8:37 AM
Forum Newbie

Forum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum Newbie

Group: General Forum Members
Last Login: Today @ 6:44 AM
Points: 3, Visits: 74
Interesting challenge as corporate infrastructure becomes more centralized and server virtualization expands. Fewer people support more platforms with automated tools. The day's of knowing or even identifying system support and admin teams seems like a distant memory.

Automated patch process in this environment is a challenge, all in the interest of cyber security and minimizing cost.
Post #1405501
Posted Thursday, January 10, 2013 8:50 AM
Valued Member

Valued MemberValued MemberValued MemberValued MemberValued MemberValued MemberValued MemberValued Member

Group: General Forum Members
Last Login: Monday, February 11, 2013 8:23 AM
Points: 50, Visits: 88
Automation is actually easy and I have no idea how they managed this mistake unless they were using a 3rd party (ie. non-MS) tool to deploy the update. We use WSUS and don't EVER have this problem because WSUS works with Windows Update specifically to install only what's applicable to that OS (System Center does the same thing but is more complex and has more features) - and includes updates for all MS products (so I can apply SQL updates along with Windows updates and minimize downtime for everyone and then only have one reboot - all without having to pay it any attention). When done properly automated update maintenance works great - I know it saves me a lot of time and manual labor.
Post #1405509
Posted Thursday, January 10, 2013 9:48 AM
SSC Eights!

SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!

Group: General Forum Members
Last Login: 2 days ago @ 8:46 AM
Points: 845, Visits: 2,331
I'd have to say that we do see security patches for SQL Server these days, and they certainly require more work for me than service packs.
SP# > current SP, install it.

MS12-070 - well, what's your current build number? And if Microsoft Updates installed the one for the lesser build number, you can't upgrade to the greater build number (or at least I haven't figured out to go from build 10.00.5512 to 10.00.5826 without getting stopped by a "you've already installed this update" error)

Last year, SQL Server 2005 had two security updates (MS12-070, MS11-049) and no service packs.
2008 had one security update (MS12-070), no service packs.
2008R2 had one security update (MS12-070), one service pack
2012 had one security update (MS12-070) and one service pack.

Sum total: two service packs, five security updates.
Post #1405530
Posted Thursday, January 10, 2013 10:10 AM


SSC-Dedicated

SSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-Dedicated

Group: Administrators
Last Login: Today @ 2:04 PM
Points: 32,780, Visits: 14,939
Nadrek (1/10/2013)

...
Sum total: two service packs, five security updates.


That's across 3 platforms, which still isnt' bad. If you look at the Oracle or DB2 lists, many more.







Follow me on Twitter: @way0utwest

Forum Etiquette: How to post data/code on a forum to get the best help
Post #1405542
Posted Thursday, January 10, 2013 10:28 AM
SSCrazy

SSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazy

Group: General Forum Members
Last Login: Monday, March 10, 2014 5:44 PM
Points: 2,225, Visits: 1,258
If you consider how hard hackers and malware developers are trying to break in and corrupt or take data or software illegally this is rather surprising. And further if you consider the complexity and diversity of SQL Server functionality, having this few "fixes" is really amazing. I know the save them up and release a number of updates at one time, but still to not have one emergency security patch after another is great.

M.


Not all gray hairs are Dinosaurs!
Post #1405550
Posted Thursday, January 10, 2013 1:54 PM
SSC-Enthusiastic

SSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-Enthusiastic

Group: General Forum Members
Last Login: Yesterday @ 9:11 AM
Points: 187, Visits: 1,817
From many rounds of patch management over the years, I would highly recommend the following:

1) Have a Computer Management Configuration Database (CMDB) in place, and make sure that the patch information is updated regularly thru an automated process.

2) If databases servers have their own CMDB, and there are good reasons to do so, make sure that the Windows support and patch management team(s) are aware of it. Provide them with the appropriate interface so they understand which are the DB servers. Not every junior Windows admin will realize that DB and web/app servers need to be treated differently.

3) SLA's for each server should be documented. It should be easy to group servers by application to see how the entire patching schedule should be set up. One of the more annoying things to deal with is spending Friday afternoon and evening re-schreschedulingxceptions to the company-wide patching window.

4) The patch coordinator for each application should be identified and easy to determine. Application inventory systems may only list the high level owner who won't always recognize 'ServerX'.

5) Make sure that you are getting the high availability from your clusters. In an active/passive setup, the passive node needs to be identified before every patching cycle. This should be automated, and fed into the process, as discussed in #1 & 2. Of course we would like every instance to be on the preferred node, but that doesn't always happen. Since cluster nodes are likely to have consecutive IP addresses and/or server names, both nodes could receive the patch at the same time, something that should obviously never happen.

The bottom line is that it is critical to have good processes in place. To quote the Yogi Berra Aflac commercial, when you don't have it, that's when you gotta have it.

Post #1405638
Posted Thursday, January 10, 2013 2:13 PM


Ten Centuries

Ten CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen Centuries

Group: General Forum Members
Last Login: Thursday, March 06, 2014 1:05 PM
Points: 1,334, Visits: 3,068
Very good point Steve! Our security team just stopped the MS13-007 patch (http://support.microsoft.com/kb/2769327) here from going onto all of our servers when it was discovered through a website scan that applying that patch to our servers would bring down the websites. They scanned our websites and found hundreds of occurrences of the REPLACE function in the .aspx code. This just goes to show you that every patch that Mickeysoft puts out is not always in your best interest. You must examine each and every one of them on a case by case basis for your shops particular situation.

"Technology is a weird thing. It brings you great gifts with one hand, and it stabs you in the back with the other. ..."
Post #1405641
Posted Thursday, January 10, 2013 2:18 PM


SSC-Dedicated

SSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-Dedicated

Group: Administrators
Last Login: Today @ 2:04 PM
Points: 32,780, Visits: 14,939
TravisDBA (1/10/2013)
Very good point Steve! Our security team just stopped the MS13-007 patch (http://support.microsoft.com/kb/2769327) here from going onto all of our servers when it was discovered through a website scan that applying that patch to our servers would bring down the websites. They scanned our websites and found hundreds of occurrences of the REPLACE function in the .aspx code. This just goes to show you that every patch that Mickeysoft puts out is not always in your best interest. You must examine each and every one of them on a case by case basis for your shops particular situation.

Good catch.

There are definitely issues with some patches. I know I'm hesitant to apply any the first month. I'd rather let someone else test things.







Follow me on Twitter: @way0utwest

Forum Etiquette: How to post data/code on a forum to get the best help
Post #1405644
Posted Thursday, January 10, 2013 4:15 PM


SSC-Addicted

SSC-AddictedSSC-AddictedSSC-AddictedSSC-AddictedSSC-AddictedSSC-AddictedSSC-AddictedSSC-Addicted

Group: General Forum Members
Last Login: Tuesday, April 01, 2014 2:57 PM
Points: 443, Visits: 822
Reading deeper into this story, it probably wasn't a patch to blame

"this was the result of Task Sequence distributed to a custom SCCM Collection. The Collection had been created/modified by an HP Engineer (adding a wildcard) and the engineer had inadvertently altered the Collection so that it was very similar in form and function to the “All Systems” Collection. The Task Sequence contained automation to format the disks"

That explains the no OS found error messages reported, and also the question of how MS OS patches could possibly be installed on the wrong systems.
Post #1405691
« Prev Topic | Next Topic »

Add to briefcase 12»»

Permissions Expand / Collapse