differentiating between ADLS and ADF and use applications

  • vega805

    Ten Centuries

    Points: 1036

    I recently completed Mike McQuillan's ADLS and USQL course on Pluralsight in addition to PragmaticWorks class on ADF v2. I really appreciate these tools and see the advantages with respect to server side ETL using SSIS (the crux of my experience).

    My question is this - when to use ADLS and ADF, the exercises in ADLS were largely importing flat files using C# operators to transform etc., but this largely seemed like one time, ad-hoc jobs, I didn’t get that this can be or  meant to be triggered, scheduled to run as a production job etc. Whereas custom pipelines in ADF can be engineered to facilitate a DWH use case for example and scheduled daily, hourly, based off another sequence etc.

    Am I missing something here? Whats the best way to differentiate? With respect to actual use cases, what is a solid ground rule for these technologies?

    _____________________________________________________________________
    As soon as you see something, you already start to intellectualize it. As soon as you intellectualize something, it is no longer what you saw. Suzuki-roshi

  • Site Owners

    SSC Guru

    Points: 80378

    Thanks for posting your issue and hopefully someone will answer soon.

    This is an automated bump to increase visibility of your question.

  • Mike McQuillan

    SSCertifiable

    Points: 5997

    Hi Vega

    Just come across your post, sorry this reply is so late. Glad you enjoyed the course!

    I go into much more detail about what can be done with U-SQL in the Stairway here on sqlservercentral.com.

    This includes how to execute jobs using PowerShell. The idea is to write your script to process files/data of some type (the data can come from databases if required - one of the Stairway articles shows how U-SQL can talk to SQL Server) on a daily/hourly/weekly/whatever schedule. Much like SSIS in fact.

    U-SQL jobs can be used in ADF as part of the pipeline, there's a Stairway article for that too.

    I'm not sure there's a solid "ground rule" I can give you for when to use each technology. I have to give the same old inconclusive answer - "it depends". For instance, I'm working for a company at the moment which runs totally on on-premise SQL Server, so I'm using SSIS there. But another place I worked at used SQL Server Azure and ingested lots of files, so it made sense to use a Data Lake there.

    Hope this helps!

    Regards,

    Mike.

     

Viewing 3 posts - 1 through 3 (of 3 total)

You must be logged in to reply to this topic. Login to reply