How to eliminate duplicate records from a database table?

SQLServerCentral is supported by Red Gate Software Ltd.

Popular Topics

Home

Members

Calendar

Who's On

Home » Article Discussions » Article Discussions by Author » Discuss content posted by rajeshwari » How to eliminate duplicate records from a...

How to eliminate duplicate records from a database table?

Rate Topic

Display Mode

Topic Options

Author

Message

rajeshwari-967105

Posted Tuesday, December 30, 2008 5:20 AM

SSC Journeyman

Group: General Forum Members
Last Login: Tuesday, September 25, 2012 1:48 AM
Points: 76, Visits: 25

Comments posted to this topic are about the item How to eliminate duplicate records from a database table?

Post #627243

Malcolm Leach

Posted Tuesday, January 06, 2009 4:01 AM

Valued Member

Group: General Forum Members
Last Login: Monday, April 25, 2011 1:09 AM
Points: 73, Visits: 68

I'm not sure I'd use this script on a table with 10 million rows in it due to the potential for the cursor to become unwieldy if there are a large number of duplicates. I have eliminated duplicates using this kind of approach before:

(you can cut and paste this entire script into QA/MS and run it with no modifications)

/* -----------------------------------------------------------------
Script by Malcolm Leach 6th Jan 2009, http://www.dbghost.com

Set up test database for demo
----------------------------------------------------------------- */

use master
go
if exists (select 1 from master..sysdatabases where name = 'EliminateDuplicatesDemo')
drop database EliminateDuplicatesDemo
go
create database EliminateDuplicatesDemo
go
use EliminateDuplicatesDemo
go
create table DupTable (id int, col1 varchar(50))
go
insert into DupTable values(1,'some text')
insert into DupTable values(1,'some text')
insert into DupTable values(2,'some text2')
insert into DupTable values(2,'some text2')
-- the ID is a duplicate on the following row but the value of col1 is not...
insert into DupTable values(2,'this is different but with the same id')
insert into DupTable values(3,'some text3')
insert into DupTable values(3,'some text3')
insert into DupTable values(4,'some text4')
go

select * from duptable

begin tran

/* -----------------------------------------------------------------
Extract all rows with duplicate ID's to a temp table
----------------------------------------------------------------- */

-- get all the duplicated ID's first
select id into TempDupIds from DupTable group by id having count(id) > 1

select distinct dt.id, dt.col1
into TempDupTableDupRows
from DupTable dt
inner join TempDupIds di on di.id = dt.id

-- this query will show up any rows that have duplicate IDs but different data
-- these rows will need "special attention"
select id from TempDupTableDupRows group by id having count(id) > 1

/* -----------------------------------------------------------------
Delete all the rows with duplicate ID's from the main table
----------------------------------------------------------------- */

delete from DupTable where id in (select id from TempDupIds)

/* -----------------------------------------------------------------
Put the rows back into the main table
----------------------------------------------------------------- */

insert into DupTable select * from TempDupTableDupRows

commit tran

-- show the cleaned up table (the problem row is still there so no data has been lost)
select * from duptable

Malcolm
DB Ghost - Build, compare and synchronize from source control = Database Change Management for SQL Server
www.dbghost.com

Post #630424

balajimu

Posted Tuesday, January 06, 2009 7:21 AM

Forum Newbie

Group: General Forum Members
Last Login: Thursday, August 13, 2009 8:30 AM
Points: 1, Visits: 14

This seems very efficient even for larger tables...... Thanks for the tips.

Post #630553

Jeff Woodard

Posted Tuesday, January 06, 2009 7:30 AM

Forum Newbie

Group: General Forum Members
Last Login: Thursday, April 23, 2009 12:30 PM
Points: 1, Visits: 17

This is a nice post - I was not aware that you could use the 'set rowcount' command to limit the number of rows deleted (or inserted or updated for that matter). In digging up a little more detail though, I came across the following note in SQL 2008 BOL. If you're designing code to work in future versions of SQL, you may want to avoid the 'set rowcount' command:

-------------- from SQL2008 BOL ---------------------------------------------------------------
Using SET ROWCOUNT will not affect DELETE, INSERT, and UPDATE statements in the next release of SQL Server. Do not use SET ROWCOUNT with DELETE, INSERT, and UPDATE statements in new development work, and plan to modify applications that currently use it. Also, for DELETE, INSERT, and UPDATE statements that currently use SET ROWCOUNT, we recommend that you rewrite them to use the TOP syntax. For more information, see DELETE (Transact-SQL), INSERT (Transact-SQL), or UPDATE (Transact-SQL).
-------------- end snippet ----------------------------------------------------------------------

Post #630563

turkbuku

Posted Tuesday, January 06, 2009 7:51 AM

Forum Newbie

Group: General Forum Members
Last Login: Friday, April 24, 2009 2:50 AM
Points: 2, Visits: 23

with Dups as (
SELECT *,row_number() over(partition by Company_name,Address order by Company_name,Address) as RowNum
FROM dbo.companies)
Delete from Dups where rownum > 1;

Post #630588

raymond lew

Posted Tuesday, January 06, 2009 10:13 AM

SSC Rookie

Group: General Forum Members
Last Login: Friday, March 27, 2009 10:01 AM
Points: 48, Visits: 123

That's awesome, turkbuku. I've use this construct (rank over blah0 with Oracle's rowid, but rowid is not available in SQL server. But this 'with' contruct works as well as rowid.

Hot damn. Thank you, man.

With a bit of cut and paste...

drop table DupTable
go
create table DupTable (id int, col1 varchar(50))
go
insert into DupTable values(1,'some text')
insert into DupTable values(1,'some text')
insert into DupTable values(2,'some text2')
insert into DupTable values(2,'some text2')
-- the ID is a duplicate on the following row but the value of col1 is not...
insert into DupTable values(2,'this is different but with the same id')
insert into DupTable values(3,'some text3')
insert into DupTable values(3,'some text3')
insert into DupTable values(4,'some text4')
GO

SELECT * FROM duptable;

with Dups as
(SELECT *,row_number() OVER
(partition by id order by id,col1) as RowNum
FROM duptable)

Delete from Dups where rownum > 1;

SELECT * FROM duptable

Post #630751

Malcolm Leach

Posted Tuesday, January 06, 2009 10:16 AM

Valued Member

Group: General Forum Members
Last Login: Monday, April 25, 2011 1:09 AM
Points: 73, Visits: 68

Cool - love it :)

Although - it still suffers from the problem of deleting potentially valuable data ;)

this solves it:

with Dups as
( SELECT *,row_number()
OVER
(partition by id,col1 order by id,col1) as RowNum
FROM duptable
)

Malcolm
DB Ghost - Build, compare and synchronize from source control = Database Change Management for SQL Server
www.dbghost.com

Post #630756

raymond lew

Posted Tuesday, January 06, 2009 10:31 AM

SSC Rookie

Group: General Forum Members
Last Login: Friday, March 27, 2009 10:01 AM
Points: 48, Visits: 123

Read all about it. Here's the link to 'Common table expressions' aka the 'with...as' construct in MS-T-SQL.

I had trouble googling for 'WITH AS'. Thought I would pass this info on.

[url=http://msdn.microsoft.com/en-us/library/ms175972.aspx][/url]

Post #630772

tzara

Posted Tuesday, January 06, 2009 3:14 PM

SSC-Enthusiastic

Group: General Forum Members
Last Login: Friday, May 17, 2013 9:00 PM
Points: 113, Visits: 54

No cursors or temp tables needed.

You can always add a primary key to a table, then remove dups.

1) using CTE statement (examples above).

or

2) delete using a self-join with a derivative table (joining back on the dup fields).

then remove the 'introduced' key, re-establish your colum(s) asserting the desired primary key.

CTE makes this operation a little easier to setup, and easier to read, more trivial. Older platforms can use derivative tables. note: also possible to use a view in a similar manner (e.g. view exposes dups) but why bother?

Post #631075

« Prev Topic | Next Topic »

Permissions