SQL Clone
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


I need to speed up the insertion of row into a table


I need to speed up the insertion of row into a table

Author
Message
Garth Zaleschuk
Garth Zaleschuk
SSC Rookie
SSC Rookie (26 reputation)SSC Rookie (26 reputation)SSC Rookie (26 reputation)SSC Rookie (26 reputation)SSC Rookie (26 reputation)SSC Rookie (26 reputation)SSC Rookie (26 reputation)SSC Rookie (26 reputation)

Group: General Forum Members
Points: 26 Visits: 39

I am in need of some tutoring and advisement. I am a nubee when it comes to designing processes as well as my dba expertise is sorely lacking. My situation is that I have a table with in excess of 500 million rows and will grow by 80+ million every month through a stored proc. The table has a primary key of 10 columns, although I could cut this down to 5 columns.

The current process is:
drop table tbl_tmp
select * into tbl_tmp
drop table tbl_mstr
select a bunch of columns(31) into tbl_mstr from tbl_tmp inner join to a current date table
alter table tbl_mstr add constraint [pk_tbl_mstr] primary key clustered (11 columns)

I am doing it this way as I need to clear all of the current year’s records and preserve all previous year’s stuff (and I don’t know any better). My initial process was to delete from where condition... current date tbl. This method seemed to take an excessive amount of time.

I thought of insert/merge/delete, but then I still had to run a delete against tbl_mstr. So I figured it still had to find the stale records amongst all those millions of rows. Again excessive time.

The box I run this on is a VM on something… but I have 4 cpus and 16gb ram running SQL Server 2016 Standard edition. My current process takes 65 minutes. I have run with 2 cpus and the time is not much different, although this leaves little cpu room for other processes.

Any advice/suggestions would be appreciated.


Steve Hall
Steve Hall
SSChampion
SSChampion (10K reputation)SSChampion (10K reputation)SSChampion (10K reputation)SSChampion (10K reputation)SSChampion (10K reputation)SSChampion (10K reputation)SSChampion (10K reputation)SSChampion (10K reputation)

Group: General Forum Members
Points: 10693 Visits: 12432
You should have a look at partitioning. IIRC as long as you have SP1 installed, you'll have access to this feature.
It took me a while to get the hang of this feature when I first encountered it, so I have a couple of articles on it:
Basic partitioning example
Autoamtic Data Purging with Partition Switching

Steve Hall
Linkedin
Blog Site
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum








































































































































































SQLServerCentral


Search