|
|
|
SSC Journeyman
      
Group: General Forum Members
Last Login: Tuesday, May 14, 2013 3:49 PM
Points: 94,
Visits: 426
|
|
Hello All, I have this situation: For several reports we need to extract about 5 millions records from a table of 124 million ( 173 gb) I am able to use the clustered index ( with a "like" statement) and the only other parameter is a column with a state so it is not helping much, the thing is that the resultant table is really big ( 5 gb ) and is taking forever to load and it is also consuming lots of resources from the server ( we usually get the error there is not enough system memory to run this query... ) That said, I need to find a way to obtain the same data in a faster way or at least not consuming that much resources, maybe in chunks but I am not sure how to do it... We have a server with 16 processors and 42 gb of ram, sql server 2006 sp3
Thanks a los
|
|
|
|
|
SSCrazy
      
Group: General Forum Members
Last Login: Tuesday, May 14, 2013 7:03 AM
Points: 2,979,
Visits: 4,389
|
|
ricardo_chicas (8/19/2010)
For several reports we need to extract about 5 millions records from a table of 124 million ( 173 gb) I am able to use the clustered index ( with a "like" statement) and the only other parameter is a column with a state so it is not helping much, the thing is that the resultant table is really big ( 5 gb ) and is taking forever to load and it is also consuming lots of resources from the server ( we usually get the error there is not enough system memory to run this query... ) That said, I need to find a way to obtain the same data in a faster way or at least not consuming that much resources, maybe in chunks but I am not sure how to do it... We have a server with 16 processors and 42 gb of ram, sql server 2006 sp3 Do you have any indexes to actually serve this query? Retrieving 5M out of 124M rows is a case where a very selective index would certainly help. On the other hand, a "like" predicate pretty much forces optimizer to do a full table scan - have you traced the query to see what is doing?
_____________________________________ Pablo (Paul) Berzukov
Author of Understanding Database Administration available at Amazon and other bookstores.
Disclaimer: Advice is provided to the best of my knowledge but no implicit or explicit warranties are provided. Since the advisor explicitly encourages testing any and all suggestions on a test non-production environment advisor should not held liable or responsible for any actions taken based on the given advice.
|
|
|
|
|
SSC Journeyman
      
Group: General Forum Members
Last Login: Tuesday, May 14, 2013 3:49 PM
Points: 94,
Visits: 426
|
|
Hi, I forgot to say that the "like" is this way: column like '123456%' so it is using the index and doing a clustered index seek
|
|
|
|
|
SSCrazy
      
Group: General Forum Members
Last Login: Tuesday, May 14, 2013 7:03 AM
Points: 2,979,
Visits: 4,389
|
|
ricardo_chicas (8/19/2010) I forgot to say that the "like" is this way: column like '123456%' so it is using the index and doing a clustered index seek
Are you positive this is what query is doing? did you check execution plan?
_____________________________________ Pablo (Paul) Berzukov
Author of Understanding Database Administration available at Amazon and other bookstores.
Disclaimer: Advice is provided to the best of my knowledge but no implicit or explicit warranties are provided. Since the advisor explicitly encourages testing any and all suggestions on a test non-production environment advisor should not held liable or responsible for any actions taken based on the given advice.
|
|
|
|
|
SSC Journeyman
      
Group: General Forum Members
Last Login: Tuesday, May 14, 2013 3:49 PM
Points: 94,
Visits: 426
|
|
| Yes, I am absolutely sure
|
|
|
|
|
SSC-Dedicated
           
Group: Administrators
Last Login: Yesterday @ 9:03 PM
Points: 31,406,
Visits: 13,723
|
|
|
|
|
|
SSC Journeyman
      
Group: General Forum Members
Last Login: Tuesday, May 14, 2013 3:49 PM
Points: 94,
Visits: 426
|
|
| I am only getting 5 columns, all of then are small, what i am looking for is a way to select chunks of that data in a optimal way....
|
|
|
|
|
SSC Veteran
      
Group: General Forum Members
Last Login: Today @ 5:31 AM
Points: 275,
Visits: 751
|
|
ricardo_chicas (8/19/2010) Hello All, I have this situation: For several reports we need to extract about 5 millions records from a table of 124 million ( 173 gb) I am able to use the clustered index ( with a "like" statement) and the only other parameter is a column with a state so it is not helping much, the thing is that the resultant table is really big ( 5 gb ) and is taking forever to load and it is also consuming lots of resources from the server ( we usually get the error there is not enough system memory to run this query... )
Exactly how are you generating the report?
SSRS ? Custom program ? T-SQL ?
Is the report generator running on the same machine as the SQL server ?
Do you really need to return all 5 million rows to the report generator?
I suppose the report generator does not print out a report with 5 million rows but instead does some aggregations of the data to print out a more reasonable number of rows. If this is the case could you perhaps do this aggregation in the source query instead of doing it in the report ?
Another possible thing to look for is if you have an ORDER BY in the source query - that might force SQL server to sort the result set which is fairly expensive for 5GB data
/SG
|
|
|
|
|
SSC-Dedicated
           
Group: Administrators
Last Login: Yesterday @ 9:03 PM
Points: 31,406,
Visits: 13,723
|
|
|
|
|
|
SSC Journeyman
      
Group: General Forum Members
Last Login: Tuesday, May 14, 2013 3:49 PM
Points: 94,
Visits: 426
|
|
Hello, It is a T-SQL program, the problem is that we really need that much info, I can't change the logic, And that info changes really often so the report must be updated every day, again, there is a way to retrieve that much info in chunk, but in an optimal way?
|
|
|
|