Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase 12»»

Insert Into temp table and then copy into 'real' table Expand / Collapse
Author
Message
Posted Thursday, September 15, 2011 6:30 AM
Grasshopper

GrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopper

Group: General Forum Members
Last Login: Monday, October 15, 2012 7:33 AM
Points: 11, Visits: 54
Hi Chaps

I have a stored procedure which is scheduled to run hourly - this stored procedure deletes all rows from an existing table and then using 'insert into' repopulates the table with data (and rebuilds indexes too).

My problem is the query which repopulates the table takes a couple of minutes to complete, during which time, because the start of the procedure includes a DELETE FROM, the table being used live is empty. Is there an alternative method which leaves the older data in tact for as long as possible until the update is complete?

My naive and uneducated mind leads me to think of maybe using the SP to insert into a temp table, then deleting from real table, then copying into real table - assuming copying from the temp to real is faster than inserting into whilst running the monster query! But I don't know how to do this.

No doubt there are also much cleaner ways to effectively update table data from a complex queries other than deleting and inserting via a stored procedures but my knowledge is pretty limited I'm ashamed to say.

I've not posted the query in the SP because it really is a monster with multiple union statements but I will if it's needed.

Cheers

Post #1175617
Posted Thursday, September 15, 2011 6:38 AM


Hall of Fame

Hall of FameHall of FameHall of FameHall of FameHall of FameHall of FameHall of FameHall of FameHall of Fame

Group: General Forum Members
Last Login: Yesterday @ 8:26 AM
Points: 3,164, Visits: 4,344
Have you looked at the MERGE statement?

I would seriously consider using this to maintain the table, instead of truncating and repopulating every time.

You could also use the huge select as part of a CTE, to be used in conjunction with said MERGE statement


____________________________________________
Space, the final frontier? not any more...
All limits henceforth are self-imposed.
“libera tute vulgaris ex”
Post #1175620
Posted Thursday, September 15, 2011 6:41 AM


SSCertifiable

SSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiable

Group: General Forum Members
Last Login: Yesterday @ 9:57 AM
Points: 6,724, Visits: 5,796
A couple of things.

1) If you're deleting everything from the destination, use TRUNCATE rather than DELETE. TRUNCATE generally runs faster because it does not fully log the rows being deleted. It just logs the extents.

2) If your query is a monster with that many UNIONs, break it up into smaller chunks so that it processes faster. Use individual INSERTS and that will give your users something to hit while the other queries are inserting.

3) If this table is so important that you can't have an "empty result set", why are you completely removing all the data from it?

4) Why aren't you doing this in the off hours if this is so important?

Yes, running the data into either a temp table or a staging table (permanent user table that can be truncated) could work for you, but there are other options.

Option A) Load Temp / Staging table, TRUNCATE / DELETE from the destination and then INSERT. (In that order)

Option B) Put an identity on the table (if there isn't one already). Load a variable with the MAX(MyIdentColumn). INSERT new data into table. DELETE FROM MyTable WHERE MyIdentColumn <= @Var.

If you absolutely must remove old data from the table, instead of doing an OUTER JOIN and just updating it with the new data, then I prefer Option B. But then, I also prefer doing this in the off hours where no one will notice a few minutes of the data doing weird things.

EDIT: FYI, I haven't used MERGE yet (we just upgraded), so all my advice is based on SQL 2k5 and below knowledge.


Brandie Tarvin, MCITP Database Administrator, MCDBA, MCSA

Webpage: http://www.BrandieTarvin.net
LiveJournal Blog: http://brandietarvin.livejournal.com/
On LinkedIn!, Google+, and Twitter.

Freelance Writer: Shadowrun
Latchkeys: Nevermore, Latchkeys: The Bootleg War, and Latchkeys: Roscoes in the Night are now available on Nook and Kindle.
Post #1175623
Posted Thursday, September 15, 2011 7:57 AM
SSC Rookie

SSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC Rookie

Group: General Forum Members
Last Login: Thursday, November 10, 2011 5:31 AM
Points: 30, Visits: 64
use merge statment, it's the best option. In other hand just serialize querys in steps, soo a job don't invoque next execution until finish one.

-
Lic. Andrés M. Aiello
DBA MSSQL - Oracle
http://aiellodba.blogspot.com/
@AndresAiello
Post #1175726
Posted Thursday, September 15, 2011 10:25 AM
Grasshopper

GrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopper

Group: General Forum Members
Last Login: Monday, October 15, 2012 7:33 AM
Points: 11, Visits: 54
The truncate and changing the union all to multiple selects has halved the time this takes to run; it's now below one minute so thank you very much for the help thus far :)

To answer one question - the data is entered regularly, every few minutes in fact, but in a format which is less than useless for reporting (just lousy software I guess!) - unfortunately the users will be running the reports all of the time and expect them to be as 'current' as possible - so an over night regime is not preferred if it can be avoided

Merge sounds like the correct approach but I've not yet fathomed that out.

Here's a snippet from the stored procedure as it is now (sorry about the messy format) - if there is anything particularly stupid that I'm missing please let me know..apart from using merge/cte etc (which would obviously need me to read a few more articles)

USE [ciswarehouse]
GO
/****** Object: StoredProcedure [dbo].[spLdd12] Script Date: 09/15/2011 17:10:17 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO

-- =============================================
-- Author: <Author,,Name>
-- Create date: <Create Date,,>
-- Description: <Description,,>
-- =============================================
ALTER PROCEDURE [dbo].[spLdd12]
AS
BEGIN
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON;

IF OBJECT_ID('MyTransformed','U') IS NOT NULL
truncate table dbo.MyTransformed

Insert into ciswarehouse.dbo.MyTransformed
(StudentRegister, Activity, Student, MarkDate, ShDate, Mark, Weekdate, Present, Absence, Authorised, Unauthorised)
SELECT dbase1.dbo.table1_studentregister.sr_id, dbase1.dbo.table1_activity.a_id, dbase1.dbo.table1_studentregister.sr_student , DATEADD(wk, n. N - 1, dbase1.dbo.table1_activity.a_start), DATEADD(dd, datediff(dd, 0, dateadd(wk, n. N - 1, dbase1.dbo.table1_activity.a_start)), 0),'/' , DATEADD(ww, DATEDIFF(ww, 0, DATEADD(wk, n. N - 1, dbase1.dbo.table1_activity.a_start)), 0), '1','0','0','0'
FROM dbase1.dbo.table1_activity INNER JOIN
dbase1.dbo.table1_studentregister ON dbase1.dbo.table1_activity.a_id = dbase1.dbo.table1_studentregister.sr_activity INNER JOIN
ciswarehouse.dbo.Tally AS n ON n. N BETWEEN 1 AND LEN(dbase1.dbo.table1_activity.a_weekpattern) AND

SUBSTRING(dbase1.dbo.table1_activity.a_weekpattern,n.N,1)='1' AND SUBSTRING(dbase1.dbo.table1_studentregister.sr_weekpattern, n. N, 1)
= '/'
WHERE (dbase1.dbo.table1_activity.a_end > '01-Aug-2011')

Insert into ciswarehouse.dbo.MyTransformed
(StudentRegister, Activity, Student, MarkDate, ShDate, Mark, Weekdate, Present, Absence, Authorised, Unauthorised)
SELECT dbase1.dbo.table1_studentregister.sr_id , dbase1.dbo.table1_activity.a_id, dbase1.dbo.table1_studentregister.sr_student , DATEADD(wk, n. N - 1,
dbase1.dbo.table1_activity.a_start), DATEADD(dd, datediff(dd, 0, dateadd(wk, n. N - 1, dbase1.dbo.table1_activity.a_start)), 0),'O' , DATEADD(ww, DATEDIFF(ww, 0, DATEADD(wk, n. N - 1, dbase1.dbo.table1_activity.a_start)), 0), '0','1','0','1'
FROM dbase1.dbo.table1_activity INNER JOIN
dbase1.dbo.table1_studentregister ON dbase1.dbo.table1_activity.a_id = dbase1.dbo.table1_studentregister.sr_activity INNER JOIN
ciswarehouse.dbo.Tally AS n ON n. N BETWEEN 1 AND LEN(dbase1.dbo.table1_activity.a_weekpattern) AND

SUBSTRING(dbase1.dbo.table1_activity.a_weekpattern,n.N,1)='1' AND SUBSTRING(dbase1.dbo.table1_studentregister.sr_weekpattern, n. N, 1)
= 'O'
WHERE (dbase1.dbo.table1_activity.a_end > '01-Aug-2011')

Insert into ciswarehouse.dbo.MyTransformed
(StudentRegister, Activity, Student, MarkDate, ShDate, Mark, Weekdate, Present, Absence, Authorised, Unauthorised)
SELECT dbase1.dbo.table1_studentregister.sr_id , dbase1.dbo.table1_activity.a_id, dbase1.dbo.table1_studentregister.sr_student , DATEADD(wk, n. N - 1,
dbase1.dbo.table1_activity.a_start), DATEADD(dd, datediff(dd, 0, dateadd(wk, n. N - 1, dbase1.dbo.table1_activity.a_start)), 0),' ' , DATEADD(ww, DATEDIFF(ww, 0, DATEADD(wk, n. N - 1, dbase1.dbo.table1_activity.a_start)), 0), '0','0','0','0'
FROM dbase1.dbo.table1_activity INNER JOIN
dbase1.dbo.table1_studentregister ON dbase1.dbo.table1_activity.a_id = dbase1.dbo.table1_studentregister.sr_activity INNER JOIN
ciswarehouse.dbo.Tally AS n ON n. N BETWEEN 1 AND LEN(dbase1.dbo.table1_activity.a_weekpattern) AND

SUBSTRING(dbase1.dbo.table1_activity.a_weekpattern,n.N,1)='1' AND SUBSTRING(dbase1.dbo.table1_studentregister.sr_weekpattern, n. N, 1)
= ' '
WHERE (dbase1.dbo.table1_activity.a_end > '01-Aug-2011')

..a bunch more selects and insert intos with slightly different WHERE clauses

alter index all on MyTransformed rebuild
end

Post #1175896
Posted Thursday, September 15, 2011 10:27 AM
Grasshopper

GrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopper

Group: General Forum Members
Last Login: Monday, October 15, 2012 7:33 AM
Points: 11, Visits: 54
Aiello DBA (9/15/2011)
use merge statment, it's the best option. In other hand just serialize querys in steps, soo a job don't invoque next execution until finish one.

-
Lic. Andrés M. Aiello
DBA MSSQL - Oracle
http://aiellodba.blogspot.com/
@AndresAiello


Hi - do you mean break the select statements into separate stored procedures and run seperate steps in the server agent or as I've done and replaced the 'union all' with multiple select/insert into ?
Post #1175897
Posted Thursday, September 15, 2011 10:39 AM
SSC Rookie

SSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC Rookie

Group: General Forum Members
Last Login: Thursday, November 10, 2011 5:31 AM
Points: 30, Visits: 64
I miss understand the problem. I think that next execution try to read, but it's the app isn it? well the best option is MERGE and you will not need to delete, because the clause "is inteligent" and delete elements that not exist anymore in temp table, update o add dependes of temp table. Are you under 2008?
-
Lic. Andrés M. Aiello
DBA MSSQL - Oracle
http://aiellodba.blogspot.com/
@AndresAiello
Post #1175906
Posted Thursday, September 15, 2011 10:44 AM
Grasshopper

GrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopper

Group: General Forum Members
Last Login: Monday, October 15, 2012 7:33 AM
Points: 11, Visits: 54
Aiello DBA (9/15/2011)
I miss understand the problem. I think that next execution try to read, but it's the app isn it? well the best option is MERGE and you will not need to delete, because the clause "is inteligent" and delete elements that not exist anymore in temp table, update o add dependes of temp table. Are you under 2008?
-
Lic. Andrés M. Aiello
DBA MSSQL - Oracle
http://aiellodba.blogspot.com/
@AndresAiello


Yeah I'm using 2008 - I had to admit I've no idea how to use Merge- atleast not yet :)
Post #1175916
Posted Thursday, September 15, 2011 10:50 AM


SSCertifiable

SSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiable

Group: General Forum Members
Last Login: Yesterday @ 9:57 AM
Points: 6,724, Visits: 5,796
l.danks (9/15/2011)
To answer one question - the data is entered regularly, every few minutes in fact, but in a format which is less than useless for reporting (just lousy software I guess!) - unfortunately the users will be running the reports all of the time and expect them to be as 'current' as possible - so an over night regime is not preferred if it can be avoided


This sounds like a case for Transactional Replication to me.


Brandie Tarvin, MCITP Database Administrator, MCDBA, MCSA

Webpage: http://www.BrandieTarvin.net
LiveJournal Blog: http://brandietarvin.livejournal.com/
On LinkedIn!, Google+, and Twitter.

Freelance Writer: Shadowrun
Latchkeys: Nevermore, Latchkeys: The Bootleg War, and Latchkeys: Roscoes in the Night are now available on Nook and Kindle.
Post #1175924
Posted Thursday, September 15, 2011 2:11 PM
SSCarpal Tunnel

SSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal Tunnel

Group: General Forum Members
Last Login: Monday, June 10, 2013 12:13 PM
Points: 4,319, Visits: 9,217
Here is how I would do this:

1) Create two tables for the process - an odd/even pair (MyTransformedDataOdd, MyTransformedDataEven).

2) Create 2 synonyms - one for the current data, one for the updated data

3) Check the day, if even - update the even table, if odd update the odd table

The shell of the process would be:

Truncate Table dbo.MyTransformedDataUpdate ...
Insert Into dbo.MyTransformedDataUpdate ...

Drop Synonym MyTransformedDataUpdate;
Drop Synonym MyTransformedDataCurrent;

If datepart(hour, getdate()) % 2 -- Odd
Begin
Create Synonym dbo.MyTransformedDataUpdate For dbo.MyTransformedDataOdd;
Create Synonym dbo.MyTransformedDataCurrent For dbo.MyTransformedDataEven;
End
Else
Create Synonym dbo.MyTransformedDataUpdate For dbo.MyTransformedDataEven;
Create Synonym dbo.MyTransformedDataCurrent For dbo.MyTransformedDataOdd;
End

The drop/create should take less than a second - but, if someone is accessing the table when it runs it will be blocked until it can switch over. You may have to put that piece in a transaction to insure someone doesn't get access while the synonyms are being swapped.

You may need to use something else to determine which tables to use - but the idea is that you always update the non actively used table and then switch the synonyms when ready. Your users would be directed to always use the 'Current' table.

Using MERGE won't help - since that will lock the table (or rows) and could cause blocking for your users as the table is being updated/inserted and I am assuming you don't want your users affected by this process.


Jeffrey Williams
Problems are opportunites brilliantly disguised as insurmountable obstacles.

How to post questions to get better answers faster
Managing Transaction Logs
Post #1176053
« Prev Topic | Next Topic »

Add to briefcase 12»»

Permissions Expand / Collapse