Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase

How to eliminate duplicate records from a database table? Expand / Collapse
Author
Message
Posted Tuesday, December 30, 2008 5:20 AM
SSC Journeyman

SSC JourneymanSSC JourneymanSSC JourneymanSSC JourneymanSSC JourneymanSSC JourneymanSSC JourneymanSSC Journeyman

Group: General Forum Members
Last Login: Tuesday, September 25, 2012 1:48 AM
Points: 76, Visits: 25
Comments posted to this topic are about the item How to eliminate duplicate records from a database table?
Post #627243
Posted Tuesday, January 6, 2009 4:01 AM
Valued Member

Valued MemberValued MemberValued MemberValued MemberValued MemberValued MemberValued MemberValued Member

Group: General Forum Members
Last Login: Tuesday, March 25, 2014 1:21 AM
Points: 73, Visits: 69
I'm not sure I'd use this script on a table with 10 million rows in it due to the potential for the cursor to become unwieldy if there are a large number of duplicates. I have eliminated duplicates using this kind of approach before:

(you can cut and paste this entire script into QA/MS and run it with no modifications)

/* -----------------------------------------------------------------
Script by Malcolm Leach 6th Jan 2009, http://www.dbghost.com

Set up test database for demo
----------------------------------------------------------------- */

use master
go
if exists (select 1 from master..sysdatabases where name = 'EliminateDuplicatesDemo')
drop database EliminateDuplicatesDemo
go
create database EliminateDuplicatesDemo
go
use EliminateDuplicatesDemo
go
create table DupTable (id int, col1 varchar(50))
go
insert into DupTable values(1,'some text')
insert into DupTable values(1,'some text')
insert into DupTable values(2,'some text2')
insert into DupTable values(2,'some text2')
-- the ID is a duplicate on the following row but the value of col1 is not...
insert into DupTable values(2,'this is different but with the same id')
insert into DupTable values(3,'some text3')
insert into DupTable values(3,'some text3')
insert into DupTable values(4,'some text4')
go

select * from duptable

begin tran

/* -----------------------------------------------------------------
Extract all rows with duplicate ID's to a temp table
----------------------------------------------------------------- */

-- get all the duplicated ID's first
select id into TempDupIds from DupTable group by id having count(id) > 1

select distinct dt.id, dt.col1
into TempDupTableDupRows
from DupTable dt
inner join TempDupIds di on di.id = dt.id

-- this query will show up any rows that have duplicate IDs but different data
-- these rows will need "special attention"
select id from TempDupTableDupRows group by id having count(id) > 1

/* -----------------------------------------------------------------
Delete all the rows with duplicate ID's from the main table
----------------------------------------------------------------- */

delete from DupTable where id in (select id from TempDupIds)

/* -----------------------------------------------------------------
Put the rows back into the main table
----------------------------------------------------------------- */

insert into DupTable select * from TempDupTableDupRows

commit tran

-- show the cleaned up table (the problem row is still there so no data has been lost)
select * from duptable


Malcolm
DB Ghost - Build, compare and synchronize from source control = Database Change Management for SQL Server
www.dbghost.com
Post #630424
Posted Tuesday, January 6, 2009 7:21 AM
Forum Newbie

Forum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum Newbie

Group: General Forum Members
Last Login: Thursday, August 13, 2009 8:30 AM
Points: 1, Visits: 14
This seems very efficient even for larger tables...... Thanks for the tips.
Post #630553
Posted Tuesday, January 6, 2009 7:30 AM
Forum Newbie

Forum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum Newbie

Group: General Forum Members
Last Login: Thursday, April 23, 2009 12:30 PM
Points: 1, Visits: 17
This is a nice post - I was not aware that you could use the 'set rowcount' command to limit the number of rows deleted (or inserted or updated for that matter). In digging up a little more detail though, I came across the following note in SQL 2008 BOL. If you're designing code to work in future versions of SQL, you may want to avoid the 'set rowcount' command:

-------------- from SQL2008 BOL ---------------------------------------------------------------
Using SET ROWCOUNT will not affect DELETE, INSERT, and UPDATE statements in the next release of SQL Server. Do not use SET ROWCOUNT with DELETE, INSERT, and UPDATE statements in new development work, and plan to modify applications that currently use it. Also, for DELETE, INSERT, and UPDATE statements that currently use SET ROWCOUNT, we recommend that you rewrite them to use the TOP syntax. For more information, see DELETE (Transact-SQL), INSERT (Transact-SQL), or UPDATE (Transact-SQL).
-------------- end snippet ----------------------------------------------------------------------
Post #630563
Posted Tuesday, January 6, 2009 7:51 AM
Forum Newbie

Forum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum Newbie

Group: General Forum Members
Last Login: Friday, April 24, 2009 2:50 AM
Points: 2, Visits: 23
with Dups as (
SELECT *,row_number() over(partition by Company_name,Address order by Company_name,Address) as RowNum
FROM dbo.companies)
Delete from Dups where rownum > 1;




Post #630588
Posted Tuesday, January 6, 2009 10:13 AM


SSC Rookie

SSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC Rookie

Group: General Forum Members
Last Login: Friday, March 27, 2009 10:01 AM
Points: 48, Visits: 123
That's awesome, turkbuku. I've use this construct (rank over blah0 with Oracle's rowid, but rowid is not available in SQL server. But this 'with' contruct works as well as rowid.

Hot damn. Thank you, man.

With a bit of cut and paste...


drop table DupTable
go
create table DupTable (id int, col1 varchar(50))
go
insert into DupTable values(1,'some text')
insert into DupTable values(1,'some text')
insert into DupTable values(2,'some text2')
insert into DupTable values(2,'some text2')
-- the ID is a duplicate on the following row but the value of col1 is not...
insert into DupTable values(2,'this is different but with the same id')
insert into DupTable values(3,'some text3')
insert into DupTable values(3,'some text3')
insert into DupTable values(4,'some text4')
GO

SELECT * FROM duptable;

with Dups as
(SELECT *,row_number() OVER
(partition by id order by id,col1) as RowNum
FROM duptable)

Delete from Dups where rownum > 1;

SELECT * FROM duptable

Post #630751
Posted Tuesday, January 6, 2009 10:16 AM
Valued Member

Valued MemberValued MemberValued MemberValued MemberValued MemberValued MemberValued MemberValued Member

Group: General Forum Members
Last Login: Tuesday, March 25, 2014 1:21 AM
Points: 73, Visits: 69
Cool - love it :)

Although - it still suffers from the problem of deleting potentially valuable data ;)

this solves it:

with Dups as
( SELECT *,row_number()
OVER
(partition by id,col1 order by id,col1) as RowNum
FROM duptable
)


Malcolm
DB Ghost - Build, compare and synchronize from source control = Database Change Management for SQL Server
www.dbghost.com
Post #630756
Posted Tuesday, January 6, 2009 10:31 AM


SSC Rookie

SSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC Rookie

Group: General Forum Members
Last Login: Friday, March 27, 2009 10:01 AM
Points: 48, Visits: 123
Read all about it. Here's the link to 'Common table expressions' aka the 'with...as' construct in MS-T-SQL.

I had trouble googling for 'WITH AS'. Thought I would pass this info on.

[url=http://msdn.microsoft.com/en-us/library/ms175972.aspx][/url]
Post #630772
Posted Tuesday, January 6, 2009 3:14 PM
SSC-Enthusiastic

SSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-Enthusiastic

Group: General Forum Members
Last Login: Monday, October 27, 2014 6:44 PM
Points: 126, Visits: 69
No cursors or temp tables needed.

You can always add a primary key to a table, then remove dups.

1) using CTE statement (examples above).

or

2) delete using a self-join with a derivative table (joining back on the dup fields).

then remove the 'introduced' key, re-establish your colum(s) asserting the desired primary key.

CTE makes this operation a little easier to setup, and easier to read, more trivial. Older platforms can use derivative tables. note: also possible to use a view in a similar manner (e.g. view exposes dups) but why bother?
Post #631075
« Prev Topic | Next Topic »

Add to briefcase

Permissions Expand / Collapse