ETL Testing Tools

  • vick12

    Right there with Babe

    Points: 796

    Hi All,

    I need to know if there is any tool which can be used to for post migration ETL testing? The ETL has been written using sql scripts to move the data from Legacy database to a new database and both have different database structures We need to test the tranformed data for accuracy and check for data quality and quantity after the ETL process is executed.

    I have questions about the following:

    1.what should be the best approach/ best practices to test the data in the new database after transformation?

    2. I would like to know if there is any tool which can be used to compare the data in old vs new database (both having different structures) ? Thanks.

  • FREDERICK

    Valued Member

    Points: 60

    Hello

    You may want to check out

    tSQLt – The Database Unit Testing Framework for SQL Server

    http://tsqlt.org/

    Tests are automatically run within transactions – this keeps tests independent and reduces any cleanup work you need

    * Tests can be grouped together within a schema – allowing you to organize your tests and use common setup methods

    * Output can be generated in plain text or XML – making it easier to integrate with a continuous integration tool

    * Provides the ability to fake tables and views, and to create stored procedure spies – allowing you to isolate the code which you are testing

    FRED

  • Orlando Colamatteo

    SSC Guru

    Points: 182207

    If you're moving to a different structure then I would not trust anything to validate that anyhow. Chances are you will need custom methods to validate your custom transformations. Common methods are reconciling row counts between systems and picking (or conjuring) a row on the source that should exercise all parts of a given transformation, e.g. 1 row on source becomes three rows on destination, and validating that you see the expected result on the destination.

    __________________________________________________________________________________________________
    There are no special teachers of virtue, because virtue is taught by the whole community. --Plato

  • Jeff Moden

    SSC Guru

    Points: 993644

    I know this is an older post but I agree with OPC.Three. If there were a testing tool that could validate your ETL results against the source, why wouldn't you just use that same testing tool to do the actual import?

    --Jeff Moden


    RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
    First step towards the paradigm shift of writing Set Based code:
    ________Stop thinking about what you want to do to a row... think, instead, of what you want to do to a column.
    "If you think its expensive to hire a professional to do the job, wait until you hire an amateur."--Red Adair
    "Change is inevitable... change for the better is not."
    When you put the right degree of spin on it, the number 3|8 is also a glyph that describes the nature of a DBAs job. 😉

    Helpful Links:
    How to post code problems

Viewing 4 posts - 1 through 4 (of 4 total)

You must be logged in to reply to this topic. Login to reply