sql query optimization for deduplication.

• Roger Jones

  Grasshopper

  Points: 10

  More actions

  June 21, 2013 at 8:59 am

  #150372

  Hi All,

  I have a query for de duplicating messages based on unique hash and then sorting and paging it, i was wondering how I can optimize it better.

  Hash and Name column in the Message table are non clustered indexes defined with storeid, folderid, Hash or Name(either)

  The only thing i see problematic is an index scan but that is same even if i remove the deduplication cte and do the main query on the Message table, i think that is more to do with select *

  Any help will be greatly appreciated.

  With DeduplicatedMessages as

  (

  select * from Message

  JOIN

  (

  SELECT MAX(MessageId) as UniqueMessageId

  FROM Message

  where StoreId=210 and FolderId=24541

  GROUP BY Hash

  ) as UniqueMessages ON

  Message.Messageid = UniqueMessages.UniqueMessageId

  )

  select * from (select ROW_NUMBER() over (order by Name ASC) as RI, * from DeduplicatedMessages) as SUB where sub.RI >= 0 and sub.RI <=1000

  Thanks

• Sean Lange

  SSC Guru

  Points: 286573

  More actions

  June 21, 2013 at 9:04 am

  #1626364

  In order to help we will need a few things:

  1. Sample DDL in the form of CREATE TABLE statements

  2. Sample data in the form of INSERT INTO statements

  3. Expected results based on the sample data

  Please take a few minutes and read the first article in my signature for best practices when posting questions.

  _______________________________________________________________

  Need help? Help us help you.

  Read the article at http://www.sqlservercentral.com/articles/Best+Practices/61537/ for best practices on asking questions.

  Need to split a string? Try Jeff Modens splitter http://www.sqlservercentral.com/articles/Tally+Table/72993/.

  Cross Tabs and Pivots, Part 1 – Converting Rows to Columns - http://www.sqlservercentral.com/articles/T-SQL/63681/
  Cross Tabs and Pivots, Part 2 - Dynamic Cross Tabs - http://www.sqlservercentral.com/articles/Crosstab/65048/
  Understanding and Using APPLY (Part 1) - http://www.sqlservercentral.com/articles/APPLY/69953/
  Understanding and Using APPLY (Part 2) - http://www.sqlservercentral.com/articles/APPLY/69954/