differentiating between ADLS and ADF and use applications

  • vega805

    Ten Centuries

    Points: 1036

    I recently completed Mike McQuillan's ADLS and USQL course on Pluralsight in addition to PragmaticWorks class on ADF v2. I really appreciate these tools and see the advantages with respect to server side ETL using SSIS (the crux of my experience).

    My question is this - when to use ADLS and ADF, the exercises in ADLS were largely importing flat files using C# operators to transform etc., but this largely seemed like one time, ad-hoc jobs, I didn’t get that this can be or  meant to be triggered, scheduled to run as a production job etc. Whereas custom pipelines in ADF can be engineered to facilitate a DWH use case for example and scheduled daily, hourly, based off another sequence etc.

    Am I missing something here? Whats the best way to differentiate? With respect to actual use cases, what is a solid ground rule for these technologies?

    As soon as you see something, you already start to intellectualize it. As soon as you intellectualize something, it is no longer what you saw. Suzuki-roshi

  • Site Owners

    SSC Guru

    Points: 80378

    Thanks for posting your issue and hopefully someone will answer soon.

    This is an automated bump to increase visibility of your question.

  • Mike McQuillan


    Points: 5997

    Hi Vega

    Just come across your post, sorry this reply is so late. Glad you enjoyed the course!

    I go into much more detail about what can be done with U-SQL in the Stairway here on sqlservercentral.com.

    This includes how to execute jobs using PowerShell. The idea is to write your script to process files/data of some type (the data can come from databases if required - one of the Stairway articles shows how U-SQL can talk to SQL Server) on a daily/hourly/weekly/whatever schedule. Much like SSIS in fact.

    U-SQL jobs can be used in ADF as part of the pipeline, there's a Stairway article for that too.

    I'm not sure there's a solid "ground rule" I can give you for when to use each technology. I have to give the same old inconclusive answer - "it depends". For instance, I'm working for a company at the moment which runs totally on on-premise SQL Server, so I'm using SSIS there. But another place I worked at used SQL Server Azure and ingested lots of files, so it made sense to use a Data Lake there.

    Hope this helps!




Viewing 3 posts - 1 through 3 (of 3 total)

You must be logged in to reply to this topic. Login to reply