differentiating between ADLS and ADF and use applications

• vega805

  Ten Centuries

  Points: 1036

  More actions

  August 14, 2019 at 4:45 am

  #3671217

  I recently completed Mike McQuillan's ADLS and USQL course on Pluralsight in addition to PragmaticWorks class on ADF v2. I really appreciate these tools and see the advantages with respect to server side ETL using SSIS (the crux of my experience).

  My question is this - when to use ADLS and ADF, the exercises in ADLS were largely importing flat files using C# operators to transform etc., but this largely seemed like one time, ad-hoc jobs, I didn’t get that this can be or  meant to be triggered, scheduled to run as a production job etc. Whereas custom pipelines in ADF can be engineered to facilitate a DWH use case for example and scheduled daily, hourly, based off another sequence etc.

  Am I missing something here? Whats the best way to differentiate? With respect to actual use cases, what is a solid ground rule for these technologies?

  _____________________________________________________________________
  As soon as you see something, you already start to intellectualize it. As soon as you intellectualize something, it is no longer what you saw. Suzuki-roshi

• Site Owners

  SSC Guru

  Points: 80342

  More actions

  August 15, 2019 at 5:10 am

  #3671476

  Thanks for posting your issue and hopefully someone will answer soon.

  This is an automated bump to increase visibility of your question.

• Mike McQuillan

  SSCertifiable

  Points: 6020

  More actions

  January 28, 2020 at 5:34 pm

  #3720985

  Hi Vega

  Just come across your post, sorry this reply is so late. Glad you enjoyed the course!

  I go into much more detail about what can be done with U-SQL in the Stairway here on sqlservercentral.com.

  This includes how to execute jobs using PowerShell. The idea is to write your script to process files/data of some type (the data can come from databases if required - one of the Stairway articles shows how U-SQL can talk to SQL Server) on a daily/hourly/weekly/whatever schedule. Much like SSIS in fact.

  U-SQL jobs can be used in ADF as part of the pipeline, there's a Stairway article for that too.

  I'm not sure there's a solid "ground rule" I can give you for when to use each technology. I have to give the same old inconclusive answer - "it depends". For instance, I'm working for a company at the moment which runs totally on on-premise SQL Server, so I'm using SSIS there. But another place I worked at used SQL Server Azure and ingested lots of files, so it made sense to use a Data Lake there.

  Hope this helps!

  Regards,

  Mike.