Home Forums SQLServerCentral.com Editorials Self-Healing ETL RE: Self-Healing ETL

  • October 12, 2018 at 10:21 am

    #2009410

    Dalkeith - Thursday, October 11, 2018 1:43 AM

    I think this would be an excellent use of AI.

    I suspect a lot in here have worked up their own tools to speed up the tedious task of matching things like columns or normalizing up or down to assist in writing necessary updates and appends. I am not completely familiar with the products out there available for this mainly because it is an infrequent task for me and it seemed as easy to write my own functions as buy an actual product.

    For me large scale ETL is still a somewhat rare request and any AI data manipulation automation is being directed at daily tasks at present.

    It's really not AI at all - it's about introducing reflection and introspection techniques into your ETL.  Programming languages have used this/made this available for some time now, and while it does take some care to know how to match things up (i.e. find the existing elements and make sure that they are still "the same") vs knowing how to detect new elements and deciding what to do with those new items, this is an old concept in a new setting.

    As Steve mentioned - while it is possible, the first question is whether you can really afford to make some guesses "on the fly":  what is the fallout of making the wrong choice/a bad matchup?  what is the performance hit?  For example - I might do this on demographic/background info (little to no risk), but I would probably lean towards "failing" the job if this were financial or regulatory data.

    ----------------------------------------------------------------------------------
    Your lack of planning does not constitute an emergency on my part...unless you're my manager...or a director and above...or a really loud-spoken end-user..All right - what was my emergency again?