Fuzzy grouping on huge data

• mawe37

  Grasshopper

  Points: 13

  More actions

  February 27, 2014 at 3:46 am

  #396338

  Hi

  We are trying to run fuzzy grouping using Business Intelligence tool of SQL Server on address field of data type nvarchar(255) for the client data. The count of records is 48,016,166. I wanted to get an idea of the max size of fuzzy grouping that anybody might have run and the approximate time taken to run the same. We are getting an error when we run the fuzzy grouping on an ‘Intel Xeon X5550 @ 2.67 GHz(2 processors)’ machine which has 24GB RAM.

  We have also tried it on 'Intel Xeon CPU E5-2650 0 @ 2.00 GHz (2 processors) which has 128 GB RAM, 16 cores but had to stop it after 48 hours as it slowed down the processing for other queries.

  Thanks

  HIM

• michal.lisinski

  SSCertifiable

  Points: 7201

  More actions

  February 27, 2014 at 5:59 am

  #1692623

  Hi

  It's generaly limited by available memory. However, there are a few things you can do to ease up on performance.

  Please convert it from varchar to char before, remove unused columns from the pipe,

  you can also minimize memory requirements running it from DTExec.exe than the SSIS Designer.

  Set max server memory, you can play with MaxMemoryUsage property.

  Br.

  Mike