It's Time to Patch and Upgrade

  • Comments posted to this topic are about the item It's Time to Patch and Upgrade

  • I posted this elsewhere in the forums yesterday:

    SQL Server Guidance to protect against speculative execution side-channel vulnerabilities
    https://support.microsoft.com/en-us/help/4073225/guidance-for-sql-server

    There have been also preliminary benchmarks of the performance hit that one of the patches cause on heavy I/O loads of 17 to 30%. Note this was PostgreSQL on Linux, but y'all should monitor your ETL and backup times. Maybe the push needed to go to SSD systems for those still on spinney disks .

    Short term we should patch and monitor.

    Long term, we really need to have an serious computer industry wide discussion on hardware and software security, programming language, processes and some of decisions made over the last two decades. There's a lot of issues that need to be unpacked.

    (As a side note, one of my main frame sysadmin buddies, is saying "I told you so" about the hardware flaws.... But IBM is also issuing patches for firmware and OSes...   :Whistling: )

  • It's also interesting that some of the browser attacks were through new enhancements designed to make browsers perform better. The fix is to disable the new features.

    Changes are a double edged sword.

    ...

    -- FORTRAN manual for Xerox Computers --

  • chrisn-585491 - Friday, January 5, 2018 6:17 AM

    (As a side note, one of my main frame sysadmin buddies, is saying "I told you so" about the hardware flaws.... )

    I wouldn't be so fast to crow if I were him. They aren't reported as vulnerable. Doesn't mean they aren't (or that there aren't equally nasty flaws waiting to be found)

    Gail Shaw
    Microsoft Certified Master: SQL Server, MVP, M.Sc (Comp Sci)
    SQL In The Wild: Discussions on DB performance with occasional diversions into recoverability

    We walk in the dark places no others will enter
    We stand on the bridge and no one may pass
  • Tuesday morning I discovered an overnight ETL process running on an Azure IaaS instance had aborted. Yesterday, I was told that Microsoft rebooted our Azure  hosted servers in the process of applying some emergency patch. I'm guessing this fix was it. While this resulted in only a couple of minutes downtime for the server, we actually lost several hours of processing work downstream. This is why I believe that fewer maintenance windows (preferably scheduled in advance) of longer duration are better than more frequent random occurrences of short duration. However, this particular issue was probably a rare event. We also need to look into making our ETL process more robust, utilizing retry logic and the capability to restart from SSIS checkpoints.

    "Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho

  • jay-h - Friday, January 5, 2018 6:45 AM

    It's also interesting that some of the browser attacks were through new enhancements designed to make browsers perform better. The fix is to disable the new features.

    Changes are a double edged sword.

    Performance optimization often times means programming shortcuts around bottlenecks. Unfortunately this sometimes means bypassing security checks or introducing new vulnerabilities to exploit.

    "Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho

  • Eric M Russell - Friday, January 5, 2018 7:23 AM

    Tuesday morning I discovered an overnight ETL process hosted on an Azure IaaS instance had aborted. Yesterday, I was told that Microsoft rebooted our Azure hosted servers in the process of applying some emergency patch. I'm guessing this was it.

    Yup, everything that my company runs/manages on Azure got rebooted. I believe it was originally scheduled for next week (when the official disclosure was supposed to happen), but careless words from AMD lead to the details of the bugs leaking early, hence the emergency patches.

    Gail Shaw
    Microsoft Certified Master: SQL Server, MVP, M.Sc (Comp Sci)
    SQL In The Wild: Discussions on DB performance with occasional diversions into recoverability

    We walk in the dark places no others will enter
    We stand on the bridge and no one may pass
  • GilaMonster - Friday, January 5, 2018 6:48 AM

    chrisn-585491 - Friday, January 5, 2018 6:17 AM

    (As a side note, one of my main frame sysadmin buddies, is saying "I told you so" about the hardware flaws.... )

    I wouldn't be so fast to crow if I were him. They aren't reported as vulnerable. Doesn't mean they aren't (or that there aren't equally nasty flaws waiting to be found)

    I edited my original post, because I double checked him and all the IBM processors have the same issues as everyone else's.

    "What we have here is a failure to lock the barn door, just because we want to milk the cows faster..." - my uncle, the dairy farmer.

  • GilaMonster - Friday, January 5, 2018 7:31 AM

    Eric M Russell - Friday, January 5, 2018 7:23 AM

    Tuesday morning I discovered an overnight ETL process hosted on an Azure IaaS instance had aborted. Yesterday, I was told that Microsoft rebooted our Azure hosted servers in the process of applying some emergency patch. I'm guessing this was it.

    Yup, everything that my company runs/manages on Azure got rebooted. I believe it was originally scheduled for next week (when the official disclosure was supposed to happen), but careless words from AMD lead to the details of the bugs leaking early, hence the emergency patches.

    Actually, folks keeping an eye on the Linux kernel development figured it out before most. Rumors have been drifting around for a few weeks.

    And several years ago, there are a few smart folks predicting this exact bug in Intel processors. (I'd have to look, it was a topic of discussion on Hacker News yesterday...)

  • chrisn-585491 - Friday, January 5, 2018 8:03 AM

    GilaMonster - Friday, January 5, 2018 7:31 AM

    Eric M Russell - Friday, January 5, 2018 7:23 AM

    Tuesday morning I discovered an overnight ETL process hosted on an Azure IaaS instance had aborted. Yesterday, I was told that Microsoft rebooted our Azure hosted servers in the process of applying some emergency patch. I'm guessing this was it.

    Yup, everything that my company runs/manages on Azure got rebooted. I believe it was originally scheduled for next week (when the official disclosure was supposed to happen), but careless words from AMD lead to the details of the bugs leaking early, hence the emergency patches.

    Actually, folks keeping an eye on the Linux kernel development figured it out before most.

    Yes, after an AMD dev merged a patch with a comment that had a great flashing neon arrow pointing to the root cause. 🙂

    Gail Shaw
    Microsoft Certified Master: SQL Server, MVP, M.Sc (Comp Sci)
    SQL In The Wild: Discussions on DB performance with occasional diversions into recoverability

    We walk in the dark places no others will enter
    We stand on the bridge and no one may pass
  • GilaMonster - Friday, January 5, 2018 8:10 AM

    chrisn-585491 - Friday, January 5, 2018 8:03 AM

    GilaMonster - Friday, January 5, 2018 7:31 AM

    Eric M Russell - Friday, January 5, 2018 7:23 AM

    Tuesday morning I discovered an overnight ETL process hosted on an Azure IaaS instance had aborted. Yesterday, I was told that Microsoft rebooted our Azure hosted servers in the process of applying some emergency patch. I'm guessing this was it.

    Yup, everything that my company runs/manages on Azure got rebooted. I believe it was originally scheduled for next week (when the official disclosure was supposed to happen), but careless words from AMD lead to the details of the bugs leaking early, hence the emergency patches.

    Actually, folks keeping an eye on the Linux kernel development figured it out before most.

    Yes, after an AMD dev merged a patch with a comment that had a great flashing neon arrow pointing to the root cause. 🙂

    Yes. :pinch:

    The good news is the two computers at the house that aren't affected are RaspberryPi. 😀

  • Things like this scare me to death.  Consider how dependent we've become on computers both large and small.  Shoot, even my pickup truck is subject to being able to be shut down remotely.  Heh... sit IoT, sit.  Good boy.

    --Jeff Moden


    RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
    First step towards the paradigm shift of writing Set Based code:
    ________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

    Change is inevitable... Change for the better is not.


    Helpful Links:
    How to post code problems
    How to Post Performance Problems
    Create a Tally Function (fnTally)

  • Jeff Moden - Friday, January 5, 2018 7:11 PM

    Things like this scare me to death.  Consider how dependent we've become on computers both large and small.  Shoot, even my pickup truck is subject to being able to be shut down remotely.  Heh... sit IoT, sit.  Good boy.

    Jeff,

    The thing is that folks in BSD community have been noticing serious issues with processors for a while (2007) and posted some serious reservations about security and Intels commitment to quality back then. But who pays attention to those paranoid kooks?!?  (Never mind others were right about the NSA monitoring everything...)

    Between the this, the massive privacy breaches over the last few years and the crappy quality of automotive code (Toyota, Ford and Volkswagen issues), the commercial hardware and software industry needs mature serious change for the better. Maybe we don't need millions of cheap IoT devices with no security? Or JavaScript applications that host 0-days. Or programming languages with buffer overflows and crappy memory management? Or CTO/CIO/CSO with no technical chops?

  • chrisn-585491 - Friday, January 5, 2018 8:35 PM

    Jeff Moden - Friday, January 5, 2018 7:11 PM

    Things like this scare me to death.  Consider how dependent we've become on computers both large and small.  Shoot, even my pickup truck is subject to being able to be shut down remotely.  Heh... sit IoT, sit.  Good boy.

    Jeff,

    The thing is that folks in BSD community have been noticing serious issues with processors for a while (2007) and posted some serious reservations about security and Intels commitment to quality back then. But who pays attention to those paranoid kooks?!?  (Never mind others were right about the NSA monitoring everything...)

    Between the this, the massive privacy breaches over the last few years and the crappy quality of automotive code (Toyota, Ford and Volkswagen issues), the commercial hardware and software industry needs mature serious change for the better. Maybe we don't need millions of cheap IoT devices with no security? Or JavaScript applications that host 0-days. Or programming languages with buffer overflows and crappy memory management? Or CTO/CIO/CSO with no technical chops?

    Heh... why anyone thought that the NSA wasn't (or currently isn't) monitoring everything is totally beyond me. 😉  But that's a subject that I'd rather leave alone for now.

    I wasn't aware of the older quality problems that you speak of because I have mostly divorced myself of the hardware for more than a decade now.  With only much more rare and insignificant problems prior to those, one of the things that you used to be able to count on (regardless of manufacturer) was the CPU itself.  It's a real shame (for me, anyway) that kind of trust has now been destroyed.

    --Jeff Moden


    RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
    First step towards the paradigm shift of writing Set Based code:
    ________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

    Change is inevitable... Change for the better is not.


    Helpful Links:
    How to post code problems
    How to Post Performance Problems
    Create a Tally Function (fnTally)

  • Jeff Moden - Friday, January 5, 2018 9:39 PM

    chrisn-585491 - Friday, January 5, 2018 8:35 PM

    Jeff Moden - Friday, January 5, 2018 7:11 PM

    I wasn't aware of the older quality problems that you speak of because I have mostly divorced myself of the hardware for more than a decade now.  With only much more rare and insignificant problems prior to those, one of the things that you used to be able to count on (regardless of manufacturer) was the CPU itself.  It's a real shame (for me, anyway) that kind of trust has now been destroyed.

    I keep an eye on hardware and other OSes since it's a life long passion and I use a variety of each outside the normal job. We need a good variety of OSes, software and hardware to have a healthy technical ecology.

Viewing 15 posts - 1 through 15 (of 17 total)

You must be logged in to reply to this topic. Login to reply