Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase

Log Files Growing, even in Simple Mode Expand / Collapse
Author
Message
Posted Thursday, February 20, 2014 5:25 PM


SSC Eights!

SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!

Group: General Forum Members
Last Login: Thursday, July 17, 2014 3:19 PM
Points: 975, Visits: 3,342
Hello Everyone
I am using a database to store a large amount of data from another SQL Database else where on the network. I am transferring data using SSIS package that executes sprocs in the destination database. I am performing a couple updates to the data once the data is in the First table. I am updating 'Null' to the NULL value in all the columns of all the tables. I found that text value in a couple of tables, so I am check them all. I have the destination database set to Simple Recover mode. However, the trans logs are growing huge, over 400Gb during the load of only about 500Gb of data. I cannot seem to figure out why. I could see it perhaps growing a little, but nothing like that.

Any suggestions, or configurations that I can check? I have looked over everything that I can think of.

Thank you in advance for all your assistance, suggestions and comments

Andrew SQLDBA
Post #1543772
Posted Thursday, February 20, 2014 5:39 PM


SSCertifiable

SSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiable

Group: General Forum Members
Last Login: Yesterday @ 6:13 PM
Points: 6,175, Visits: 7,254
Just to clarify...

You're updating 500 gigs of data and wondering why you have growth of 400 gigs, or did I misunderstand that part?

In SSIS, on your destination, what's your commit size/Rows per Batch? If it's blank/2147483647, then your entire load is going in as a single transaction. Cut that up a bit. :)



- Craig Farrell

Never stop learning, even if it hurts. Ego bruises are practically mandatory as you learn unless you've never risked enough to make a mistake.

For better assistance in answering your questions | Forum Netiquette
For index/tuning help, follow these directions. |Tally Tables

Twitter: @AnyWayDBA
Post #1543776
Posted Thursday, February 20, 2014 6:11 PM


SSC Eights!

SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!

Group: General Forum Members
Last Login: Thursday, July 17, 2014 3:19 PM
Points: 975, Visits: 3,342
Thank You, very good advice on that.
Yes, you understand correctly. I would really like the database to not log anything for this data load.

Yes, I had left that to the default value of:
Rows Per Batch = blank
Maximum Insert Commit Size = 2147483647

I am going to change the values, test some more, and see what works the best for that one step. Only one of the tables that I am pumping data from is large, over 1.5 billion rows. All the others are really small in row count.

Thank you Greatly.

Andrew SQLDBA
Post #1543778
Posted Thursday, February 20, 2014 6:48 PM


SSCoach

SSCoachSSCoachSSCoachSSCoachSSCoachSSCoachSSCoachSSCoachSSCoachSSCoachSSCoach

Group: General Forum Members
Last Login: Yesterday @ 12:07 PM
Points: 15,562, Visits: 27,940
No way in the world, at all, to avoid the log, so best practice is to find ways to deal with it. No other choice there. If you don't need recover to a point in time, you might try bulk-logged. It can reduce log size for some operations, but, it will require log backups, unlike Simple.

----------------------------------------------------
"The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood..." Theodore Roosevelt
The Scary DBA
Author of: SQL Server 2012 Query Performance Tuning
SQL Server 2008 Query Performance Tuning Distilled
and
SQL Server Execution Plans

Product Evangelist for Red Gate Software
Post #1543785
Posted Thursday, February 20, 2014 9:52 PM


SSC-Insane

SSC-InsaneSSC-InsaneSSC-InsaneSSC-InsaneSSC-InsaneSSC-InsaneSSC-InsaneSSC-InsaneSSC-InsaneSSC-InsaneSSC-Insane

Group: General Forum Members
Last Login: Yesterday @ 9:07 PM
Points: 21,351, Visits: 15,032
Grant Fritchey (2/20/2014)
No way in the world, at all, to avoid the log, so best practice is to find ways to deal with it. No other choice there. If you don't need recover to a point in time, you might try bulk-logged. It can reduce log size for some operations, but, it will require log backups, unlike Simple.


Like Grant said, there is no way to avoid a hit to your log. By reducing the number of records per batch you could lessen that impact and improve overall performance.




Jason AKA CirqueDeSQLeil
I have given a name to my pain...
MCM SQL Server


SQL RNNR

Posting Performance Based Questions - Gail Shaw
Posting Data Etiquette - Jeff Moden
Hidden RBAR - Jeff Moden
VLFs and the Tran Log - Kimberly Tripp
Post #1543810
Posted Friday, February 21, 2014 12:10 AM


SSC-Forever

SSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-ForeverSSC-Forever

Group: General Forum Members
Last Login: Yesterday @ 3:21 PM
Points: 42,495, Visits: 35,566
AndrewSQLDBA (2/20/2014)
I would really like the database to not log anything for this data load.


You'd like a failure anywhere in the load to cause the destination DB to be marked suspect and need to be restored from backup? That's what you're asking when you ask for unlogged operations (which don't exist)

You can change the batch size, you can see if it's possible to get the SSIS to run BULK INSERT or similar, which can be minimally logged, but SQL does not run any unlogged operations because if it did, a single error (pk violation, fk violation, data type mismatch, etc) would result in a suspect DB because it would have no way to roll back part of the operation.



Gail Shaw
Microsoft Certified Master: SQL Server 2008, MVP
SQL In The Wild: Discussions on DB performance with occasional diversions into recoverability

We walk in the dark places no others will enter
We stand on the bridge and no one may pass

Post #1543845
Posted Friday, February 21, 2014 3:39 AM


SSC Eights!

SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!

Group: General Forum Members
Last Login: Thursday, July 17, 2014 3:19 PM
Points: 975, Visits: 3,342
Thank You Everyone
That is what I knew. I was just hoping that someone more intelligent that I would know of something else.

Thanks
Andrew SQLDBA
Post #1543897
« Prev Topic | Next Topic »

Add to briefcase

Permissions Expand / Collapse