Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase

ETL Testing Tools Expand / Collapse
Author
Message
Posted Tuesday, October 23, 2012 7:48 AM
SSC Rookie

SSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC Rookie

Group: General Forum Members
Last Login: Tuesday, November 20, 2012 9:04 PM
Points: 28, Visits: 86
Hi All,

I need to know if there is any tool which can be used to for post migration ETL testing? The ETL has been written using sql scripts to move the data from Legacy database to a new database and both have different database structures We need to test the tranformed data for accuracy and check for data quality and quantity after the ETL process is executed.
I have questions about the following:
1.what should be the best approach/ best practices to test the data in the new database after transformation?
2. I would like to know if there is any tool which can be used to compare the data in old vs new database (both having different structures) ? Thanks.
Post #1376052
Posted Saturday, February 9, 2013 12:20 PM
Forum Newbie

Forum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum Newbie

Group: General Forum Members
Last Login: Wednesday, September 25, 2013 2:01 PM
Points: 4, Visits: 29
Hello

You may want to check out
tSQLt – The Database Unit Testing Framework for SQL Server
http://tsqlt.org/

Tests are automatically run within transactions – this keeps tests independent and reduces any cleanup work you need

* Tests can be grouped together within a schema – allowing you to organize your tests and use common setup methods

* Output can be generated in plain text or XML – making it easier to integrate with a continuous integration tool

* Provides the ability to fake tables and views, and to create stored procedure spies – allowing you to isolate the code which you are testing


FRED



Post #1418041
Posted Saturday, February 9, 2013 2:02 PM


SSCertifiable

SSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiable

Group: General Forum Members
Last Login: Today @ 10:47 AM
Points: 7,126, Visits: 12,727
If you're moving to a different structure then I would not trust anything to validate that anyhow. Chances are you will need custom methods to validate your custom transformations. Common methods are reconciling row counts between systems and picking (or conjuring) a row on the source that should exercise all parts of a given transformation, e.g. 1 row on source becomes three rows on destination, and validating that you see the expected result on the destination.

__________________________________________________________________________________________________
There are no special teachers of virtue, because virtue is taught by the whole community. --Plato
Post #1418048
Posted Thursday, September 4, 2014 7:44 PM


SSC-Dedicated

SSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-Dedicated

Group: General Forum Members
Last Login: Today @ 6:30 PM
Points: 35,370, Visits: 31,911
I know this is an older post but I agree with OPC.Three. If there were a testing tool that could validate your ETL results against the source, why wouldn't you just use that same testing tool to do the actual import?

--Jeff Moden
"RBAR is pronounced "ree-bar" and is a "Modenism" for "Row-By-Agonizing-Row".

First step towards the paradigm shift of writing Set Based code:
Stop thinking about what you want to do to a row... think, instead, of what you want to do to a column."

(play on words) "Just because you CAN do something in T-SQL, doesn't mean you SHOULDN'T." --22 Aug 2013

Helpful Links:
How to post code problems
How to post performance problems
Post #1610718
« Prev Topic | Next Topic »

Add to briefcase

Permissions Expand / Collapse