Metadata Driven Pipelines – Data Engineering with Fabric

  • Comments posted to this topic are about the item Metadata Driven Pipelines – Data Engineering with Fabric

    John F. Miner III
    Microsoft Data Platform MVP
    a.k.a. - The Crafty DBA

  • You might be interested in what we have done with our ADF Add-On product, ChillETL.  It is completely metadata driven, and works with any ADF pipeline parameters.

  • This was removed by the editor as SPAM

  • I am attempting the same pattern, but pulling from on-prem SQL tables and inserting into Tables instead of Files in the LakeHouse.  Thank you for sharing everything you have, it has been very helpful.  The last issue I'm running into is a generic error message when using variables for the destination source and table.  Have you attempted destination tables instead of files?


    [font="Tahoma"]Personal blog relating fishing to database administration:[/font]

    [font="Comic Sans MS"]https://davegugg.wordpress.com[/url]/[/font]

  • Please post the error message to this article.  I will take a look.  Delta tables have acid properties.  They support select, insert, update and delete actions.  So this is possible but I have not tried it.  I land everything into raw as files and use SPARK notebooks for data engineering.

    John F. Miner III
    Microsoft Data Platform MVP
    a.k.a. - The Crafty DBA

  • I was able to figure it out by seeing a support thread where someone else had asked a similar question.  The issue wasn't the tables, it was that I had made the destination Lakehouse into a variable.  When doing that, you have to use the GUID of the Lakehouse, not its name.  Hope that info may help anyone else who is looking.  Thanks!


    [font="Tahoma"]Personal blog relating fishing to database administration:[/font]

    [font="Comic Sans MS"]https://davegugg.wordpress.com[/url]/[/font]

Viewing 6 posts - 1 through 5 (of 5 total)

You must be logged in to reply to this topic. Login to reply