• Jeff Moden - Saturday, November 17, 2018 12:59 PM

    xsevensinzx - Friday, November 16, 2018 8:39 PM

    You have Polybase...
    ...{snip}...
    where all you need to focus on is the INSERT and NEVER THE UPDATE.

    Now THAT's an interesting idea!  I'm going to have to look into that and you should write an article on it. 😀

    Always lack the time to do so from a technical standpoint others would want. Maybe I should just write something less technical in terms of syntax and just talk about it to start. I can certainly try.

    But yes, the idea here is something I do a lot here and influence from Kimball's books when he talks a lot about doing as much of the transformation on disk as possible. If you treat your data lake storage or document store as your physical hard disks, then the idea is the same. Do as much in the document store as possible and then find a way to link the final output to your data warehouse. In the case of SQL Server, that link or window as Microsoft calls it, is Polybase. It's an amazing piece of technology that many are not really utilizing to it's fullest. You can basically have the power of hundreds of computers behind your SMP system. Then of course, reduce the amount of UPDATES and massive CPU sucks your warehouse is doing.

    This is why I had said in the past, I will likely always use document stores (NoSQL) with all my future projects. It's just too damn powerful not to use it.