Data Orchestration

  • Steve Jones - SSC Editor

    SSC Guru

    Points: 719974

    Comments posted to this topic are about the item Data Orchestration

  • roger.plowman

    SSChampion

    Points: 10243

    Playing devil's advocate for a moment, perhaps we're nearing the point of public backlash. They are already starting to come down hard on data being collected, it wouldn't take much for them to overreact and then companies will find themselves unable to pick and choose which data they collect.

    GDPR is a worrying first sign of this backlash. Given that, perhaps the ever-growing data problem might correct itself...

  • skeleton567

    SSCertifiable

    Points: 5050

    "I've been thinking that with SQL Server 2019 we will start to access data where it lives, not move it to another place we want it. To me, this is more of what future data orchestration might involve. I ran across an article that takes a slightly different approach, thinking AI and other products will help better move data around, and perhaps that's true, but I do think more and more we want to query data where it lives, and use larger, distributed compute platforms to do this."

    Wow, for all of your sakes, I really truely hope you don't have to go back to the days of transactional data and analytical data residing together. I've been advocating for the opposite for years.  I see the two as competing for resources and conceptually incompatible.

    I agree that there is definitely an art to data orchestration, but see a large part of that exactly why data architects are so vital.  I would understand them being mixed in a start-up shop where data quantities and users have not grown so much yet.  Especially with the poorly designed transactions and poorly designed queries that I've seen it seems there is a large risk to both.

     

     

    Rick

    The only thing worse than being an influencer
    is believing one.

  • skeleton567

    SSCertifiable

    Points: 5050

    Roger, while I in some ways am also against the massive collection and retention of data just because you can, I really don't expect the trend to change much, GDPR or not.  It just won't be so obvious.  I just rejected a new version of software I've used for decades because it now requires that I connect to internet each time I use it.  Ain't gonna happen.

    Another thing I have long advocated is that companies stop even allowing direct internet access to their data stores, instead only allowing internet access to stand-alone systems with totally identity-purged data available.  Data would be moved to and from internal data stores only across internal connections.

    Sure, it's more complicated, but also much more safe.  Instead of waiting for the public backlash, we need to anticipate it and handle it.

    • This reply was modified 4 months, 1 week ago by  skeleton567.

    Rick

    The only thing worse than being an influencer
    is believing one.

  • Jeff Moden

    SSC Guru

    Points: 996661

    skeleton567 wrote:

    I just rejected a new version of software I've used for decades because it now requires that I connect to internet each time I use it.  Ain't gonna happen.

    A million likes on this one, Rick!

    --Jeff Moden


    RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
    First step towards the paradigm shift of writing Set Based code:
    ________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.
    "Change is inevitable... change for the better is not".
    "If "pre-optimization" is the root of all evil, then what does the resulting no optimization lead to?"

    Helpful Links:
    How to post code problems
    How to Post Performance Problems
    Create a Tally Function (fnTally)

Viewing 5 posts - 1 through 5 (of 5 total)

You must be logged in to reply to this topic. Login to reply