Pairing Data Vault and Biml for DW Agility

  • benjamin.seidle

    SSC-Addicted

    Points: 498

    Comments posted to this topic are about the item Pairing Data Vault and Biml for DW Agility

  • mark 66658

    SSC Journeyman

    Points: 93

    Hi, thanks for the article on DataVault and BIML! I'd very much like to see a series of articles on the topic if at all possible. A more 'in depth' study if you will. I've done a fair bit of reading but have not implemented this in practice. Would love to be able to learn more.

  • Koen Verbeeck

    SSC Guru

    Points: 258890

    This is a blog post, not a technical article.

    Nice story, but I expected at least some scripts...

    I hope this is an introduction to a great series about the combination of Data Vault and BIML.

    Need an answer? No, you need a question
    My blog at https://sqlkover.com.
    MCSE Business Intelligence - Microsoft Data Platform MVP

  • Phil Parkin

    SSC Guru

    Points: 243481

    mark 66658 (9/3/2014)


    Hi, thanks for the article on DataVault and BIML! I'd very much like to see a series of articles on the topic if at all possible. A more 'in depth' study if you will. I've done a fair bit of reading but have not implemented this in practice. Would love to be able to learn more.

    Agreed – this article is both interesting and a tease!:-) Please write more.

    If the answer to your question can be found with a brief Google search, please perform the search yourself, rather than expecting one of the SSC members to do it for you.

  • hennie7863

    SSCommitted

    Points: 1543

    Please more information. I'm interested in the technical implementation..

  • Knut Boehnert

    SSCrazy

    Points: 2946

    Interesting teaser post. I am working on a similar project currently to rewrite our Data Warehouse and Benjamin made a good point to show that DV is not a good platform for reporting - this needs to be addressed separately.

    Technically a DV is pretty much box standard. Each hub and link have their own unique surrogate ID on which you hang and track the business key. The concept is what makes a DV interesting. Instead of combining keys and data into one table you put keys into one table (hub or links) and data into others (satellites). So data is separated from keys. This basic core concept is what alienates lots of people.

    The other core concept of DV is "All the data all the time". So no matter whether the data is good or bad or ugly you just load it into the DV and worry about the quality later. This gives the DV a power to be a repository of all the business data. Some other system creates data for invoices that are not in the original source system? Just load it into a new satellite. How to get it into reports? That is a different kettle of fish.

    Data Vault is not so much a technical scripting. It is more about the design concept that a new person needs to understand. One very nice description can be found on Hans Hultgren's site (http://hanshultgren.files.wordpress.com/2013/09/data-vault-modeling-guide-2013-v2.pdf). To make an analogy running some scripts to implement a DV is like taking a car out of for a road trip without filling up the tank - you will get somewhere but your mileage will definitely vary and arrival where you want to be is for sure very questionable.

  • Koen Verbeeck

    SSC Guru

    Points: 258890

    Knut Boehnert (9/4/2014)


    Interesting teaser post. I am working on a similar project currently to rewrite our Data Warehouse and Benjamin made a good point to show that DV is not a good platform for reporting - this needs to be addressed separately.

    Indeed. DV is more of a system of record, after which data marts are created for reporting purposes.

    Knut Boehnert (9/4/2014)


    The other core concept of DV is "All the data all the time".

    I believe the DV community is now more believing in a "All of the data in scope, all of the time" 😀

    Knut Boehnert (9/4/2014)


    Data Vault is not so much a technical scripting. It is more about the design concept that a new person needs to understand. ... . To make an analogy running some scripts to implement a DV is like taking a car out of for a road trip without filling up the tank - you will get somewhere but your mileage will definitely vary and arrival where you want to be is for sure very questionable.

    True, but the DV community usually boasts how easily DV can be generated (at least the Raw layer), so that's why I was very curious to see some BIML scripts to generate DV objects. Quite the dissapointment... 🙂

    Need an answer? No, you need a question
    My blog at https://sqlkover.com.
    MCSE Business Intelligence - Microsoft Data Platform MVP

  • enders-437268

    SSC Enthusiast

    Points: 155

    For the non technicals, could explain how many tables were involved ? How long did it take to learn BIML ? Had any pre experience with other programing languages ? How often do you have to change the BIML scripts ?

    For the techies do you use a staging database ? What is the load speed ? How many rows ? How many columns ? Is it Datavault 1.0 or 2.0 ? How do you do mappings ? How do you handle errors ? Does biml generate the tables in datavault or reads it from datavault ? Do you create SSAS cubes ? How do you implement those cubes ? As views on the Datavault or physical tables ?

    Just a few questions

  • andycao

    Say Hey Kid

    Points: 698

    what ender said! +1

  • Thomas LeBlanc

    SSCertifiable

    Points: 7733

    Great description.

    But, you need to give some examples and scripts and diagrams to make this a great article.

    Now, you got me wanting to read the book, but I probably won't until I see tangible examples to get me to gravitate to a Data Vault.

    Thomas

    Thomas LeBlanc, MVP Data Platform Consultant

  • Data Wrangler

    SSC Enthusiast

    Points: 150

    I too thought that this was the intro to an in-depth article. While the post may inspire us to consider new ways to design and implement a data warehouse, it would be most helpful to get details, such as trade offs made, roadblocks encountered, what you would do differently next time, and most importantly for this community, some technical implementation details. Please write more!

    Jonathan Woodard

  • benjamin.seidle

    SSC-Addicted

    Points: 498

    Thanks for the warm reception! More articles and details to follow soon. Meantime if you have any questions on specifics feel free to hit me up.

  • benjamin.seidle

    SSC-Addicted

    Points: 498

    Beaten to the punch! Pretty good presentation from the BIML experts on how to use BIML to generate Data Vault Models

    https://www.youtube.com/watch?v=QVS79-YcPsA

  • Koen Verbeeck

    SSC Guru

    Points: 258890

    benjamin.seidle (12/10/2014)


    Beaten to the punch! Pretty good presentation from the BIML experts on how to use BIML to generate Data Vault Models

    https://www.youtube.com/watch?v=QVS79-YcPsA%5B/quote%5D

    Thanks for the link!

    Need an answer? No, you need a question
    My blog at https://sqlkover.com.
    MCSE Business Intelligence - Microsoft Data Platform MVP

  • hennie7863

    SSCommitted

    Points: 1543

    Interesting...Thnx

Viewing 15 posts - 1 through 15 (of 15 total)

You must be logged in to reply to this topic. Login to reply