What's the difference between the Data Reconciliation layer and ETL processes?

Question

Post reply

What's the difference between the Data Reconciliation layer and ETL processes?

akirajt

Right there with Babe

Points: 762
More actions
April 1, 2012 at 5:57 am

#254062

What is the difference between the Data Reconciliation layer and ETL processes? They look identical to me.
Another question: Is "stage database" a part of ETL or the Data Reconciliation layer?

Viewing 8 posts - 1 through 8 (of 8 total)

You must be logged in to reply to this topic. Login to reply

Raunak Jhawar SSCoach Points: 15701 More actions · Answer 1

Using ETL tools a BI professional can integrate, reconciliate and clean data. 🙂

Raunak J

Raunak Jhawar SSCoach Points: 15701 More actions · Answer 2

There should really be nothing as "stage database". It should be stage tables.

Think of them as an area where the BI professional would perform all the cleaning, integrating and reconciliating of the data before loading them into actual tables/cubes for end user reporting/analysis needs.

Raunak J

Andy Hyslop SSCrazy Eights Points: 9198 More actions · Answer 3

There should really be nothing as "stage database". It should be stage tables

It depends I've seen warehouses where the staging tables are contained within the main DW and also (my personal preference) where the staging tables are contained within a staging DB and the DW tables contained within a separate DB, this keeps things cleaner IMO

Andy

==========================================================================================================================
A computer lets you make more mistakes faster than any invention in human history - with the possible exceptions of handguns and tequila. Mitch Ratcliffe

MMartin1 One Orange Chip Points: 27850 More actions · Answer 4

From the diagram, the ETL brings in your data from your own operation system(s) and from the outside world. This would be a good place to make sure the data from the outside world is your own so I guess that is where the use of reconcile comes in.

----------------------------------------------------

Koen Verbeeck SSC Guru Points: 259165 More actions · Answer 5

Raunak Jhawar (4/2/2012)
There should really be nothing as "stage database". It should be stage tables.

And if you put the staging tables in a database you have a ...?

Need an answer? No, you need a question
My blog at https://sqlkover.com.
MCSE Business Intelligence - Microsoft Data Platform MVP

Raunak Jhawar SSCoach Points: 15701 More actions · Answer 6

Koen Verbeeck (4/4/2012)
Raunak Jhawar (4/2/2012)
There should really be nothing as "stage database". It should be stage tables.
And if you put the staging tables in a database you have a ...?

Stumped 🙂

Raunak J

Eric M Russell SSC Guru Points: 125596 More actions · Answer 7

Data Reconciliation can involve things like identifying the uniqueness, completeness, and validity of records. That logic may not be part of the injest, but rather a process that analyzes and updates the data after it's been moved to the intermediate staging tables. I would consider it part of the transform step.

Extract (injest into staging), Transform (reconciliation), Load (into data marts)

"Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho