July 27, 2016 at 12:15 pm
Guys -
Curious what you have done or seen with large data sets.
I have 4 tables the largest of which is 26 Billion rows.
I am partitioning the 4 tables into Yearly partitions based on the date read of the data in each. so IE: 2010, 2011 etc..
Once the data moves past 120 days the data can no longer be altered or changed and is simply archival data for reporting.
To create the functions to support the ETL of 120 days and then move the breaking wave of data to archive do I:
Have 4 tables with 120 days using triggers that then move the data to the partitioned tables? So Primary1,2 etc then after 120 days archive1,2, etc...
Or continue to use the single partitioned table and base it off date with some other functions for faster etl on just the 120 days?
The reason I need to do this is 2 sided, 1 is for speed of queries and views and reporting with data marts etc...
The second is the ETL process of merging with a smaller set of data and updates that represent just the 120 days it can be changed.
Any insights or ideas around this are welcome. I realize this is an open sided question and appreciate any insights you guys may have.
Viewing 0 posts
You must be logged in to reply to this topic. Login to reply