SQLServerCentral / Discuss Content Posted by Charles Hawkins / Article Discussions / Article Discussions by Author / Deduping Data in SQL Server 2005 / Latest Posts

RE: Deduping Data in SQL Server 2005

André Cardoso — Wed, 07 Feb 2007 04:45:00 GMT

Another option (that works with SQL 2000) and even in cases of all the columns having the same value (no column to differentiate the rows), is inserting the result set into a new table with an identity column (or adding an identity column to the original table). After that, it's just a matter of keeping the distinct rows as shown in the comments.

I can't remember where I read this solution, but it was either here in SQL Server Central or SQL Team foruns.

André Cardoso

RE: Deduping Data in SQL Server 2005

USKiwi — Tue, 07 Feb 2006 16:30:00 GMT

Interesting demonstration of the ROW_NUMBER() function. Please say it ain’t so, Joe! - that you are not using cursors to remove duplicate rows. Even the technique of a SELECT DISTINCT into a temporary table would be a better option. As other readers have commented, there are a number of ways to remove duplicate rows. This would be my approach:

DELETE Employee

FROM Employee a INNER JOIN (SELECT Empid,

FName,

LName,

MIN(RefDate) AS 'MinDate'

FROM Employee

GROUP BY Empid, FName, LName) b

ON a.Empid = b.Empid

AND a.FName = b.FName

AND a.LName = b.LName

AND a.RefDate > b.MinDate

This would still leave the issue of James verses Jim that would need to be resolved separately. If you didn’t care about spelling variations and wanted to assume that the first entry was the correct one then this would work:

DELETE Employee

FROM Employee a INNER JOIN (SELECT Empid,

MIN(RefDate) AS 'MinDate'

FROM Employee

GROUP BY Empid) b

ON a.Empid = b.Empid

AND a.RefDate > b.MinDate

I would be interested in the question of performance between the two techniques but I’d put my money on mine which I suspect has a whole lot less overhead even as a cross join than having the engine generate a row position.

RE: Deduping Data in SQL Server 2005

sscbm21 — Tue, 07 Feb 2006 13:14:00 GMT

great examples! If only duplicates need to be removed the ROW_NUMBER() may not be needed.

WITH cteEmployeeOrderedByMyRank AS(SELECT RANK() OVER (PARTITION BY EMPID,FNAME,LNAME ORDER BY REFDATE ASC) AS PartitionRank, * FROM EMPLOYEE -- WHERE 1 = 1)DELETE FROM cteEmployeeOrderedByMyRankWHERE PartitionRank > 1 ; It surely seems to be much faster than the cursor based apporach.

RE: Deduping Data in SQL Server 2005

Aries Manlig — Tue, 07 Feb 2006 09:24:00 GMT

Is there a performance gain to using this function and technique? Or is it just one of those "hey cool function --- let's use it"?

RE: Deduping Data in SQL Server 2005

Heiko Hatzfeld — Tue, 07 Feb 2006 08:11:00 GMT

Hi...

I usually clean up my "mess" with "something" like this

delete from myTable where myID NOT in

(select min(MyID) from myTable group by myUniqueField[s])

But I think the best application for CTE are recursive querries...

RE: Deduping Data in SQL Server 2005

ALZDBA — Tue, 07 Feb 2006 04:03:00 GMT

nice example

Here is another :

WITH cteEmployeeOrderedByMyRowNumber AS(SELECT ROW_NUMBER() OVER (ORDER BY EMPID ASC, REFDATE ASC) AS MyRowNumber, Row_NUMBER() Over (Partition By EMPID,FNAME,LNAME Order By REFDATE ASC) as PartitionRank, * FROM EMPLOYEE -- WHERE 1 = 1)DELETE FROM cteEmployeeOrderedByMyRowNumber where PartitionRank > 1 ;

just to get it in the tips of the fingers

RE: Deduping Data in SQL Server 2005

drnetwork — Tue, 07 Feb 2006 03:37:00 GMT

When you are populating the record set, take out the GO between the next-to-the-last and the last insert statements. Having this penultimate GO in the set of queries will remove the scope of the @NOW variable and cause the last INSERT to fail.

Deduping Data in SQL Server 2005

drnetwork — Mon, 30 Jan 2006 17:35:00 GMT

Comments posted to this topic are about the content posted at http://www.sqlservercentral.com/columnists/chawkins/dedupingdatainsqlserver2005.asp