|
|
|
Grasshopper
      
Group: General Forum Members
Last Login: Monday, October 15, 2012 7:33 AM
Points: 11,
Visits: 54
|
|
Hi Chaps
I have a stored procedure which is scheduled to run hourly - this stored procedure deletes all rows from an existing table and then using 'insert into' repopulates the table with data (and rebuilds indexes too).
My problem is the query which repopulates the table takes a couple of minutes to complete, during which time, because the start of the procedure includes a DELETE FROM, the table being used live is empty. Is there an alternative method which leaves the older data in tact for as long as possible until the update is complete?
My naive and uneducated mind leads me to think of maybe using the SP to insert into a temp table, then deleting from real table, then copying into real table - assuming copying from the temp to real is faster than inserting into whilst running the monster query! But I don't know how to do this.
No doubt there are also much cleaner ways to effectively update table data from a complex queries other than deleting and inserting via a stored procedures but my knowledge is pretty limited I'm ashamed to say.
I've not posted the query in the SP because it really is a monster with multiple union statements but I will if it's needed.
Cheers
|
|
|
|
|
Hall of Fame
       
Group: General Forum Members
Last Login: Yesterday @ 8:26 AM
Points: 3,164,
Visits: 4,344
|
|
Have you looked at the MERGE statement?
I would seriously consider using this to maintain the table, instead of truncating and repopulating every time.
You could also use the huge select as part of a CTE, to be used in conjunction with said MERGE statement
____________________________________________ Space, the final frontier? not any more... All limits henceforth are self-imposed. “libera tute vulgaris ex”
|
|
|
|
|
SSCertifiable
       
Group: General Forum Members
Last Login: Yesterday @ 9:57 AM
Points: 6,724,
Visits: 5,796
|
|
A couple of things.
1) If you're deleting everything from the destination, use TRUNCATE rather than DELETE. TRUNCATE generally runs faster because it does not fully log the rows being deleted. It just logs the extents.
2) If your query is a monster with that many UNIONs, break it up into smaller chunks so that it processes faster. Use individual INSERTS and that will give your users something to hit while the other queries are inserting.
3) If this table is so important that you can't have an "empty result set", why are you completely removing all the data from it?
4) Why aren't you doing this in the off hours if this is so important?
Yes, running the data into either a temp table or a staging table (permanent user table that can be truncated) could work for you, but there are other options.
Option A) Load Temp / Staging table, TRUNCATE / DELETE from the destination and then INSERT. (In that order)
Option B) Put an identity on the table (if there isn't one already). Load a variable with the MAX(MyIdentColumn). INSERT new data into table. DELETE FROM MyTable WHERE MyIdentColumn <= @Var.
If you absolutely must remove old data from the table, instead of doing an OUTER JOIN and just updating it with the new data, then I prefer Option B. But then, I also prefer doing this in the off hours where no one will notice a few minutes of the data doing weird things.
EDIT: FYI, I haven't used MERGE yet (we just upgraded), so all my advice is based on SQL 2k5 and below knowledge.
Brandie Tarvin, MCITP Database Administrator, MCDBA, MCSA
Webpage: http://www.BrandieTarvin.net LiveJournal Blog: http://brandietarvin.livejournal.com/ On LinkedIn!, Google+, and Twitter.
Freelance Writer: Shadowrun Latchkeys: Nevermore, Latchkeys: The Bootleg War, and Latchkeys: Roscoes in the Night are now available on Nook and Kindle.
|
|
|
|
|
SSC Rookie
      
Group: General Forum Members
Last Login: Thursday, November 10, 2011 5:31 AM
Points: 30,
Visits: 64
|
|
use merge statment, it's the best option. In other hand just serialize querys in steps, soo a job don't invoque next execution until finish one.
- Lic. Andrés M. Aiello DBA MSSQL - Oracle http://aiellodba.blogspot.com/ @AndresAiello
|
|
|
|
|
Grasshopper
      
Group: General Forum Members
Last Login: Monday, October 15, 2012 7:33 AM
Points: 11,
Visits: 54
|
|
The truncate and changing the union all to multiple selects has halved the time this takes to run; it's now below one minute so thank you very much for the help thus far :)
To answer one question - the data is entered regularly, every few minutes in fact, but in a format which is less than useless for reporting (just lousy software I guess!) - unfortunately the users will be running the reports all of the time and expect them to be as 'current' as possible - so an over night regime is not preferred if it can be avoided
Merge sounds like the correct approach but I've not yet fathomed that out.
Here's a snippet from the stored procedure as it is now (sorry about the messy format) - if there is anything particularly stupid that I'm missing please let me know..apart from using merge/cte etc (which would obviously need me to read a few more articles)
USE [ciswarehouse] GO /****** Object: StoredProcedure [dbo].[spLdd12] Script Date: 09/15/2011 17:10:17 ******/ SET ANSI_NULLS ON GO SET QUOTED_IDENTIFIER ON GO
-- ============================================= -- Author: <Author,,Name> -- Create date: <Create Date,,> -- Description: <Description,,> -- ============================================= ALTER PROCEDURE [dbo].[spLdd12] AS BEGIN -- SET NOCOUNT ON added to prevent extra result sets from -- interfering with SELECT statements. SET NOCOUNT ON;
IF OBJECT_ID('MyTransformed','U') IS NOT NULL truncate table dbo.MyTransformed
Insert into ciswarehouse.dbo.MyTransformed (StudentRegister, Activity, Student, MarkDate, ShDate, Mark, Weekdate, Present, Absence, Authorised, Unauthorised) SELECT dbase1.dbo.table1_studentregister.sr_id, dbase1.dbo.table1_activity.a_id, dbase1.dbo.table1_studentregister.sr_student , DATEADD(wk, n. N - 1, dbase1.dbo.table1_activity.a_start), DATEADD(dd, datediff(dd, 0, dateadd(wk, n. N - 1, dbase1.dbo.table1_activity.a_start)), 0),'/' , DATEADD(ww, DATEDIFF(ww, 0, DATEADD(wk, n. N - 1, dbase1.dbo.table1_activity.a_start)), 0), '1','0','0','0' FROM dbase1.dbo.table1_activity INNER JOIN dbase1.dbo.table1_studentregister ON dbase1.dbo.table1_activity.a_id = dbase1.dbo.table1_studentregister.sr_activity INNER JOIN ciswarehouse.dbo.Tally AS n ON n. N BETWEEN 1 AND LEN(dbase1.dbo.table1_activity.a_weekpattern) AND
SUBSTRING(dbase1.dbo.table1_activity.a_weekpattern,n.N,1)='1' AND SUBSTRING(dbase1.dbo.table1_studentregister.sr_weekpattern, n. N, 1) = '/' WHERE (dbase1.dbo.table1_activity.a_end > '01-Aug-2011')
Insert into ciswarehouse.dbo.MyTransformed (StudentRegister, Activity, Student, MarkDate, ShDate, Mark, Weekdate, Present, Absence, Authorised, Unauthorised) SELECT dbase1.dbo.table1_studentregister.sr_id , dbase1.dbo.table1_activity.a_id, dbase1.dbo.table1_studentregister.sr_student , DATEADD(wk, n. N - 1, dbase1.dbo.table1_activity.a_start), DATEADD(dd, datediff(dd, 0, dateadd(wk, n. N - 1, dbase1.dbo.table1_activity.a_start)), 0),'O' , DATEADD(ww, DATEDIFF(ww, 0, DATEADD(wk, n. N - 1, dbase1.dbo.table1_activity.a_start)), 0), '0','1','0','1' FROM dbase1.dbo.table1_activity INNER JOIN dbase1.dbo.table1_studentregister ON dbase1.dbo.table1_activity.a_id = dbase1.dbo.table1_studentregister.sr_activity INNER JOIN ciswarehouse.dbo.Tally AS n ON n. N BETWEEN 1 AND LEN(dbase1.dbo.table1_activity.a_weekpattern) AND
SUBSTRING(dbase1.dbo.table1_activity.a_weekpattern,n.N,1)='1' AND SUBSTRING(dbase1.dbo.table1_studentregister.sr_weekpattern, n. N, 1) = 'O' WHERE (dbase1.dbo.table1_activity.a_end > '01-Aug-2011')
Insert into ciswarehouse.dbo.MyTransformed (StudentRegister, Activity, Student, MarkDate, ShDate, Mark, Weekdate, Present, Absence, Authorised, Unauthorised) SELECT dbase1.dbo.table1_studentregister.sr_id , dbase1.dbo.table1_activity.a_id, dbase1.dbo.table1_studentregister.sr_student , DATEADD(wk, n. N - 1, dbase1.dbo.table1_activity.a_start), DATEADD(dd, datediff(dd, 0, dateadd(wk, n. N - 1, dbase1.dbo.table1_activity.a_start)), 0),' ' , DATEADD(ww, DATEDIFF(ww, 0, DATEADD(wk, n. N - 1, dbase1.dbo.table1_activity.a_start)), 0), '0','0','0','0' FROM dbase1.dbo.table1_activity INNER JOIN dbase1.dbo.table1_studentregister ON dbase1.dbo.table1_activity.a_id = dbase1.dbo.table1_studentregister.sr_activity INNER JOIN ciswarehouse.dbo.Tally AS n ON n. N BETWEEN 1 AND LEN(dbase1.dbo.table1_activity.a_weekpattern) AND
SUBSTRING(dbase1.dbo.table1_activity.a_weekpattern,n.N,1)='1' AND SUBSTRING(dbase1.dbo.table1_studentregister.sr_weekpattern, n. N, 1) = ' ' WHERE (dbase1.dbo.table1_activity.a_end > '01-Aug-2011')
..a bunch more selects and insert intos with slightly different WHERE clauses
alter index all on MyTransformed rebuild end
|
|
|
|
|
Grasshopper
      
Group: General Forum Members
Last Login: Monday, October 15, 2012 7:33 AM
Points: 11,
Visits: 54
|
|
Aiello DBA (9/15/2011) use merge statment, it's the best option. In other hand just serialize querys in steps, soo a job don't invoque next execution until finish one.
- Lic. Andrés M. Aiello DBA MSSQL - Oracle http://aiellodba.blogspot.com/ @AndresAiello
Hi - do you mean break the select statements into separate stored procedures and run seperate steps in the server agent or as I've done and replaced the 'union all' with multiple select/insert into ?
|
|
|
|
|
SSC Rookie
      
Group: General Forum Members
Last Login: Thursday, November 10, 2011 5:31 AM
Points: 30,
Visits: 64
|
|
I miss understand the problem. I think that next execution try to read, but it's the app isn it? well the best option is MERGE and you will not need to delete, because the clause "is inteligent" and delete elements that not exist anymore in temp table, update o add dependes of temp table. Are you under 2008? - Lic. Andrés M. Aiello DBA MSSQL - Oracle http://aiellodba.blogspot.com/ @AndresAiello
|
|
|
|
|
Grasshopper
      
Group: General Forum Members
Last Login: Monday, October 15, 2012 7:33 AM
Points: 11,
Visits: 54
|
|
Aiello DBA (9/15/2011) I miss understand the problem. I think that next execution try to read, but it's the app isn it? well the best option is MERGE and you will not need to delete, because the clause "is inteligent" and delete elements that not exist anymore in temp table, update o add dependes of temp table. Are you under 2008? - Lic. Andrés M. Aiello DBA MSSQL - Oracle http://aiellodba.blogspot.com/ @AndresAiello
Yeah I'm using 2008 - I had to admit I've no idea how to use Merge- atleast not yet :)
|
|
|
|
|
SSCertifiable
       
Group: General Forum Members
Last Login: Yesterday @ 9:57 AM
Points: 6,724,
Visits: 5,796
|
|
l.danks (9/15/2011) To answer one question - the data is entered regularly, every few minutes in fact, but in a format which is less than useless for reporting (just lousy software I guess!) - unfortunately the users will be running the reports all of the time and expect them to be as 'current' as possible - so an over night regime is not preferred if it can be avoided
This sounds like a case for Transactional Replication to me.
Brandie Tarvin, MCITP Database Administrator, MCDBA, MCSA
Webpage: http://www.BrandieTarvin.net LiveJournal Blog: http://brandietarvin.livejournal.com/ On LinkedIn!, Google+, and Twitter.
Freelance Writer: Shadowrun Latchkeys: Nevermore, Latchkeys: The Bootleg War, and Latchkeys: Roscoes in the Night are now available on Nook and Kindle.
|
|
|
|
|
SSCarpal Tunnel
       
Group: General Forum Members
Last Login: Monday, June 10, 2013 12:13 PM
Points: 4,319,
Visits: 9,217
|
|
Here is how I would do this:
1) Create two tables for the process - an odd/even pair (MyTransformedDataOdd, MyTransformedDataEven).
2) Create 2 synonyms - one for the current data, one for the updated data
3) Check the day, if even - update the even table, if odd update the odd table
The shell of the process would be:
Truncate Table dbo.MyTransformedDataUpdate ... Insert Into dbo.MyTransformedDataUpdate ...
Drop Synonym MyTransformedDataUpdate; Drop Synonym MyTransformedDataCurrent;
If datepart(hour, getdate()) % 2 -- Odd Begin Create Synonym dbo.MyTransformedDataUpdate For dbo.MyTransformedDataOdd; Create Synonym dbo.MyTransformedDataCurrent For dbo.MyTransformedDataEven; End Else Create Synonym dbo.MyTransformedDataUpdate For dbo.MyTransformedDataEven; Create Synonym dbo.MyTransformedDataCurrent For dbo.MyTransformedDataOdd; End
The drop/create should take less than a second - but, if someone is accessing the table when it runs it will be blocked until it can switch over. You may have to put that piece in a transaction to insure someone doesn't get access while the synonyms are being swapped.
You may need to use something else to determine which tables to use - but the idea is that you always update the non actively used table and then switch the synonyms when ready. Your users would be directed to always use the 'Current' table.
Using MERGE won't help - since that will lock the table (or rows) and could cause blocking for your users as the table is being updated/inserted and I am assuming you don't want your users affected by this process.
Jeffrey Williams Problems are opportunites brilliantly disguised as insurmountable obstacles.
How to post questions to get better answers faster Managing Transaction Logs
|
|
|
|