• henryKrinkle (10/4/2016)


    D1, T1, C1, P1, ORDER1, LINE1, 100

    D1, T2, C1, P1, ORDER1, LINE1, 500

    D1, T3, C1, P1, ORDER1, LINE1, 100

    Hi,

    Thanks for your input.

    In an ideal world, yes, the lines would increment. This is a reflection of what my source system allows - it literally duplicates the line number. Adding my own line number would be okay but if I got anything wrong at ETL and ended up having duplicates in the fact table, it may be difficult to troubleshoot.

    Henry

    Thus why you should deploy any solution to a test environment first, to try to catch those unknowns. In a normal line of work, an ETL gathering data from an outside source should have (I say should though it probably does not happen much in the real world) a document describing in technical detail the data rows, columns and formats that you should expect. It is like a service agreement. An ETL may fail but if you can point to the document and state the reason was non-adherence by the outside data, then the reason out of your scope and you are in the clear.

    ----------------------------------------------------