SQL Server Integrates Hadoop and Spark out-of-the box: The Why?

  • Frank Banin

    Ten Centuries

    Points: 1323

    Comments posted to this topic are about the item SQL Server Integrates Hadoop and Spark out-of-the box: The Why?

    Frank Banin
    BI and Advanced Analytics Professional.

  • Chris Clements

    Grasshopper

    Points: 13

    Fantastic information here, but there were so many grammar errors that they distracted me during the reading.  Sorry...not trying to be a jerk here, but thought it needed to be pointed out for the future.


    Have you ever imagined a world without hypothetical situations?

  • Jeff Moden

    SSC Guru

    Points: 994938

    Absolutely fascinating article... especially the "in a nutshell" history and "The WHY" as to what's happening with 2019.  Looks like I have a lot of reading to do thanks to all of the links you provided.

    It also explains a recent surge in questions on Spark SQL.  A lot of people are trying to apply what they know about T-SQL (and other flavors of SQL) to Spark and failing because they don't realize that SQL <> SQL.  For example, the DATEDIFF function in Spark SQL is relatively crippled in comparison to the T-SQL version and so cannot be used in the same manner for much.  However, if people take the time to lookup the relatively good documentation on the various functions in Spark SQL, they'd find a wealth of computational power in different functions that can (for example) greatly simplify such things as the computations that we use DATEDIFF in T-SQL for but in Spark SQL.  It's a powerful "SQL" (in my "first blush" examination of the documentation) but it's a different "SQL".  Knowing that up front will greatly reduce the anxieties of learning a different flavor of SQL.

    Thanks again for the article and "well done"!  I'm looking forward to reading the articles at the other end of all the links.

    --Jeff Moden


    RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
    First step towards the paradigm shift of writing Set Based code:
    ________Stop thinking about what you want to do to a row... think, instead, of what you want to do to a column.
    "If you think its expensive to hire a professional to do the job, wait until you hire an amateur."--Red Adair
    "Change is inevitable... change for the better is not."
    When you put the right degree of spin on it, the number 3|8 is also a glyph that describes the nature of a DBAs job. 😉

    Helpful Links:
    How to post code problems
    Create a Tally Function (fnTally)

Viewing 3 posts - 1 through 3 (of 3 total)

You must be logged in to reply to this topic. Login to reply