February 26, 2013 at 6:48 am
Hi all,
We are about to migrate one of our core data repositories (a Data mart of Person & Organization contact details) into a new version.
The data model of the new version is quite different from the existing one and the data is being migrated via a SSIS Package. We have done several trial runs using this package and the data "looks good" , but I would like to be confident that everything was migrated properly.
I have caught some obvious errors by running simple COUNT(*) scripts against the newly populated tables, but because the new model is more normalized than the existing one, there won't be a 1:1 match of record counts between the old and the new model for many of the tables.
I searched through 'Scripts' and 'Articles' here at SSC and found some good data profiling scripts, but I wonder if anyone can recommend an article/book that provides a sort of "migration checklist", best practice approach, etc?
I appreciate any thoughts that you can provide.
Steve
February 26, 2013 at 7:13 am
You will need to generate a control totals script. For your data of person and organization details you can have a for example:
1) Number of person records migrated.
2) Number of organisation records.
3) Number of persons in each department
4) Sum of all dates of birth.
You will want these control totals rules written in plain simple English and then go about developing scripts to generate these numbers on the source system AND also create scripts which will produce the same totals on the new system
Once the migration is run and the control totals scripts is run, if both sets of figures don’t add up, then you have a problem!
A useful check for number of person records, I use to convert dates to YYYYMMDD format and perform a sum on all dates. If the sum of this value doesn't match then either some person records didn't get migrated or there was a conversion error in the DOB field etc.
---------------------------------------------------------
It takes a minimal capacity for rational thought to see that the corporate 'free press' is a structurally irrational and biased, and extremely violent, system of elite propaganda.
David Edwards - Media lens[/url]
Society has varying and conflicting interests; what is called objectivity is the disguise of one of these interests - that of neutrality. But neutrality is a fiction in an unneutral world. There are victims, there are executioners, and there are bystanders... and the 'objectivity' of the bystander calls for inaction while other heads fall.
Howard Zinn
February 26, 2013 at 7:17 am
Interesting!
Thank you Abu.
Viewing 3 posts - 1 through 2 (of 2 total)
You must be logged in to reply to this topic. Login to reply