I recently completed Mike McQuillan's ADLS and USQL course on Pluralsight in addition to PragmaticWorks class on ADF v2. I really appreciate these tools and see the advantages with respect to server side ETL using SSIS (the crux of my experience).
My question is this - when to use ADLS and ADF, the exercises in ADLS were largely importing flat files using C# operators to transform etc., but this largely seemed like one time, ad-hoc jobs, I didn’t get that this can be or meant to be triggered, scheduled to run as a production job etc. Whereas custom pipelines in ADF can be engineered to facilitate a DWH use case for example and scheduled daily, hourly, based off another sequence etc.
Am I missing something here? Whats the best way to differentiate? With respect to actual use cases, what is a solid ground rule for these technologies?
As soon as you see something, you already start to intellectualize it. As soon as you intellectualize something, it is no longer what you saw. Suzuki-roshi