using SSIS to perform data validation in ETL app

  • Phil Parkin

    SSC Guru

    Points: 244798

    jreece (6/22/2009)


    John,

    --

    I am now going to add a few derived columns - I thought "Create_Date" and "Change_Date" might be very useful for tracking updates.

    You might also consider doing these at the database level. Your CreateDate field just needs a DefaultValue of GetDate() and ChangeDate needs an UPDATE trigger to keep it maintained ...

    If the answer to your question can be found with a brief Google search, please perform the search yourself, rather than expecting one of the SSC members to do it for you.
    See https://www.sqlservercentral.com/articles/forum-etiquette-how-to-post-datacode-on-a-forum-to-get-the-best-help/ for details of how to post T-SQL code-related questions.

  • John Rowan

    SSC Guru

    Points: 56440

    Phil Parkin (6/22/2009)


    jreece (6/22/2009)


    John,

    --

    I am now going to add a few derived columns - I thought "Create_Date" and "Change_Date" might be very useful for tracking updates.

    You might also consider doing these at the database level. Your CreateDate field just needs a DefaultValue of GetDate() and ChangeDate needs an UPDATE trigger to keep it maintained ...

    I think that depends on what you are after. I routinely use column defaults on CreateDate columns in OLTP systems. For OLAP or data warehouse DBs, I typically use a LoadDate column when loading data and I assign the LoadDate value from a variable in the SSIS package that gets its value at SSIS startup time so that everything that was loaded as part of the SSIS package run has the same LoadDate.

    John Rowan

    ======================================================
    ======================================================
    Forum Etiquette: How to post data/code on a forum to get the best help[/url] - by Jeff Moden

  • jreece

    Ten Centuries

    Points: 1138

    That's exactly what I want to do, tag all the records loaded with the same date stamp so I can identify them by batch. Thanks again - I can see it will take me a long time to get up to speed with this tool. I don't even know what I don't know. (sorry to get all Yogi Berra on you)

  • Deepmala

    Newbie

    Points: 1

    ?Well I have always used Daton's SQL connector  for my Warehousing. You can add any Data Source here in a single place

  • Phil Parkin

    SSC Guru

    Points: 244798

    Deepmala wrote:

    ?Well I have always used Daton's SQL connector  for my Warehousing. You can add any Data Source here in a single place

    Unless you are going to justify this comment by describing exactly how this product performs data validation, I am going to have to mark your comment as spam.

    If the answer to your question can be found with a brief Google search, please perform the search yourself, rather than expecting one of the SSC members to do it for you.
    See https://www.sqlservercentral.com/articles/forum-etiquette-how-to-post-datacode-on-a-forum-to-get-the-best-help/ for details of how to post T-SQL code-related questions.

Viewing 5 posts - 16 through 20 (of 20 total)

You must be logged in to reply to this topic. Login to reply