I have ran out of ideas on the following problem
.Maybe someone would be able to help me out.I have a package to remove duplicate records from 1 table and move the unique records to a new table .Below is the flow of my SSIS package
Oledb source = 1 table [2 million records]
Transformation =Fuzzy grouping on [based on 6 columns] [score is 70%]
The min similarity of each column is 0.7%
Conditional Split = Unique & Duplicate
Oledb destination =2 tables [Unique & Duplicate table ]
Source table ---> Fuzzy Grouping---->Conditional Split ---->Unique
My Server information is as belowMicrosoft SQL Server 2005 - 9.00.3042.00 SP2
Windows Server 2003 SP2
My Problem is , My package execution goes on for 2 continuous days and still is at [Fuzzy Grouping Inner Data Flow] Progress: Finding similar records - 78 percent complete.
1)Is it suppose to take so long to complete ?
2)How can I improve the performance ?
3)Upon checking, the memory used by the process is 400,000 K
There is no other application or process running on my server except for this package.It take up to 3 hours to complete 5%
.Do help me out as I am not able to go on like this for 2 days and still not completed.This package is to run on every month end.
Thank you in advance.
“I haven't failed, I've found 10,000 ways that don't work”........Thomas Alva Edison