February 20, 2014 at 5:25 pm
Hello Everyone
I am using a database to store a large amount of data from another SQL Database else where on the network. I am transferring data using SSIS package that executes sprocs in the destination database. I am performing a couple updates to the data once the data is in the First table. I am updating 'Null' to the NULL value in all the columns of all the tables. I found that text value in a couple of tables, so I am check them all. I have the destination database set to Simple Recover mode. However, the trans logs are growing huge, over 400Gb during the load of only about 500Gb of data. I cannot seem to figure out why. I could see it perhaps growing a little, but nothing like that.
Any suggestions, or configurations that I can check? I have looked over everything that I can think of.
Thank you in advance for all your assistance, suggestions and comments
Andrew SQLDBA
February 20, 2014 at 5:39 pm
Just to clarify...
You're updating 500 gigs of data and wondering why you have growth of 400 gigs, or did I misunderstand that part?
In SSIS, on your destination, what's your commit size/Rows per Batch? If it's blank/2147483647, then your entire load is going in as a single transaction. Cut that up a bit. 🙂
Never stop learning, even if it hurts. Ego bruises are practically mandatory as you learn unless you've never risked enough to make a mistake.
For better assistance in answering your questions[/url] | Forum Netiquette
For index/tuning help, follow these directions.[/url] |Tally Tables[/url]
Twitter: @AnyWayDBA
February 20, 2014 at 6:11 pm
Thank You, very good advice on that.
Yes, you understand correctly. I would really like the database to not log anything for this data load.
Yes, I had left that to the default value of:
Rows Per Batch = blank
Maximum Insert Commit Size = 2147483647
I am going to change the values, test some more, and see what works the best for that one step. Only one of the tables that I am pumping data from is large, over 1.5 billion rows. All the others are really small in row count.
Thank you Greatly.
Andrew SQLDBA
February 20, 2014 at 6:48 pm
No way in the world, at all, to avoid the log, so best practice is to find ways to deal with it. No other choice there. If you don't need recover to a point in time, you might try bulk-logged. It can reduce log size for some operations, but, it will require log backups, unlike Simple.
"The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood"
- Theodore Roosevelt
Author of:
SQL Server Execution Plans
SQL Server Query Performance Tuning
February 20, 2014 at 9:52 pm
Grant Fritchey (2/20/2014)
No way in the world, at all, to avoid the log, so best practice is to find ways to deal with it. No other choice there. If you don't need recover to a point in time, you might try bulk-logged. It can reduce log size for some operations, but, it will require log backups, unlike Simple.
Like Grant said, there is no way to avoid a hit to your log. By reducing the number of records per batch you could lessen that impact and improve overall performance.
Jason...AKA CirqueDeSQLeil
_______________________________________________
I have given a name to my pain...MCM SQL Server, MVP
SQL RNNR
Posting Performance Based Questions - Gail Shaw[/url]
Learn Extended Events
February 21, 2014 at 12:10 am
AndrewSQLDBA (2/20/2014)
I would really like the database to not log anything for this data load.
You'd like a failure anywhere in the load to cause the destination DB to be marked suspect and need to be restored from backup? That's what you're asking when you ask for unlogged operations (which don't exist)
You can change the batch size, you can see if it's possible to get the SSIS to run BULK INSERT or similar, which can be minimally logged, but SQL does not run any unlogged operations because if it did, a single error (pk violation, fk violation, data type mismatch, etc) would result in a suspect DB because it would have no way to roll back part of the operation.
Gail Shaw
Microsoft Certified Master: SQL Server, MVP, M.Sc (Comp Sci)
SQL In The Wild: Discussions on DB performance with occasional diversions into recoverability
February 21, 2014 at 3:39 am
Thank You Everyone
That is what I knew. I was just hoping that someone more intelligent that I would know of something else.
Thanks
Andrew SQLDBA
Viewing 7 posts - 1 through 6 (of 6 total)
You must be logged in to reply to this topic. Login to reply