SQL Clone
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


Log Files Growing, even in Simple Mode


Log Files Growing, even in Simple Mode

Author
Message
AndrewSQLDBA
AndrewSQLDBA
SSCommitted
SSCommitted (1.8K reputation)SSCommitted (1.8K reputation)SSCommitted (1.8K reputation)SSCommitted (1.8K reputation)SSCommitted (1.8K reputation)SSCommitted (1.8K reputation)SSCommitted (1.8K reputation)SSCommitted (1.8K reputation)

Group: General Forum Members
Points: 1752 Visits: 3427
Hello Everyone
I am using a database to store a large amount of data from another SQL Database else where on the network. I am transferring data using SSIS package that executes sprocs in the destination database. I am performing a couple updates to the data once the data is in the First table. I am updating 'Null' to the NULL value in all the columns of all the tables. I found that text value in a couple of tables, so I am check them all. I have the destination database set to Simple Recover mode. However, the trans logs are growing huge, over 400Gb during the load of only about 500Gb of data. I cannot seem to figure out why. I could see it perhaps growing a little, but nothing like that.

Any suggestions, or configurations that I can check? I have looked over everything that I can think of.

Thank you in advance for all your assistance, suggestions and comments

Andrew SQLDBA
Evil Kraig F
Evil Kraig F
SSCrazy Eights
SSCrazy Eights (8.5K reputation)SSCrazy Eights (8.5K reputation)SSCrazy Eights (8.5K reputation)SSCrazy Eights (8.5K reputation)SSCrazy Eights (8.5K reputation)SSCrazy Eights (8.5K reputation)SSCrazy Eights (8.5K reputation)SSCrazy Eights (8.5K reputation)

Group: General Forum Members
Points: 8531 Visits: 7660
Just to clarify...

You're updating 500 gigs of data and wondering why you have growth of 400 gigs, or did I misunderstand that part?

In SSIS, on your destination, what's your commit size/Rows per Batch? If it's blank/2147483647, then your entire load is going in as a single transaction. Cut that up a bit. Smile


- Craig Farrell

Never stop learning, even if it hurts. Ego bruises are practically mandatory as you learn unless you've never risked enough to make a mistake.

For better assistance in answering your questions | Forum Netiquette
For index/tuning help, follow these directions. |Tally Tables

Twitter: @AnyWayDBA
AndrewSQLDBA
AndrewSQLDBA
SSCommitted
SSCommitted (1.8K reputation)SSCommitted (1.8K reputation)SSCommitted (1.8K reputation)SSCommitted (1.8K reputation)SSCommitted (1.8K reputation)SSCommitted (1.8K reputation)SSCommitted (1.8K reputation)SSCommitted (1.8K reputation)

Group: General Forum Members
Points: 1752 Visits: 3427
Thank You, very good advice on that.
Yes, you understand correctly. I would really like the database to not log anything for this data load.

Yes, I had left that to the default value of:
Rows Per Batch = blank
Maximum Insert Commit Size = 2147483647

I am going to change the values, test some more, and see what works the best for that one step. Only one of the tables that I am pumping data from is large, over 1.5 billion rows. All the others are really small in row count.

Thank you Greatly.

Andrew SQLDBA
Grant Fritchey
Grant Fritchey
SSC-Dedicated
SSC-Dedicated (38K reputation)SSC-Dedicated (38K reputation)SSC-Dedicated (38K reputation)SSC-Dedicated (38K reputation)SSC-Dedicated (38K reputation)SSC-Dedicated (38K reputation)SSC-Dedicated (38K reputation)SSC-Dedicated (38K reputation)

Group: General Forum Members
Points: 38953 Visits: 32616
No way in the world, at all, to avoid the log, so best practice is to find ways to deal with it. No other choice there. If you don't need recover to a point in time, you might try bulk-logged. It can reduce log size for some operations, but, it will require log backups, unlike Simple.

----------------------------------------------------
The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood...
Theodore Roosevelt

The Scary DBA
Author of: SQL Server Query Performance Tuning and SQL Server Execution Plans
Product Evangelist for Red Gate Software
SQLRNNR
SQLRNNR
SSC-Dedicated
SSC-Dedicated (31K reputation)SSC-Dedicated (31K reputation)SSC-Dedicated (31K reputation)SSC-Dedicated (31K reputation)SSC-Dedicated (31K reputation)SSC-Dedicated (31K reputation)SSC-Dedicated (31K reputation)SSC-Dedicated (31K reputation)

Group: General Forum Members
Points: 31746 Visits: 18550
Grant Fritchey (2/20/2014)
No way in the world, at all, to avoid the log, so best practice is to find ways to deal with it. No other choice there. If you don't need recover to a point in time, you might try bulk-logged. It can reduce log size for some operations, but, it will require log backups, unlike Simple.


Like Grant said, there is no way to avoid a hit to your log. By reducing the number of records per batch you could lessen that impact and improve overall performance.



Jason AKA CirqueDeSQLeil
I have given a name to my pain...
MCM SQL Server, MVP


SQL RNNR

Posting Performance Based Questions - Gail Shaw

GilaMonster
GilaMonster
SSC Guru
SSC Guru (86K reputation)SSC Guru (86K reputation)SSC Guru (86K reputation)SSC Guru (86K reputation)SSC Guru (86K reputation)SSC Guru (86K reputation)SSC Guru (86K reputation)SSC Guru (86K reputation)

Group: General Forum Members
Points: 86024 Visits: 45221
AndrewSQLDBA (2/20/2014)
I would really like the database to not log anything for this data load.


You'd like a failure anywhere in the load to cause the destination DB to be marked suspect and need to be restored from backup? That's what you're asking when you ask for unlogged operations (which don't exist)

You can change the batch size, you can see if it's possible to get the SSIS to run BULK INSERT or similar, which can be minimally logged, but SQL does not run any unlogged operations because if it did, a single error (pk violation, fk violation, data type mismatch, etc) would result in a suspect DB because it would have no way to roll back part of the operation.

Gail Shaw
Microsoft Certified Master: SQL Server, MVP, M.Sc (Comp Sci)
SQL In The Wild: Discussions on DB performance with occasional diversions into recoverability

We walk in the dark places no others will enter
We stand on the bridge and no one may pass


AndrewSQLDBA
AndrewSQLDBA
SSCommitted
SSCommitted (1.8K reputation)SSCommitted (1.8K reputation)SSCommitted (1.8K reputation)SSCommitted (1.8K reputation)SSCommitted (1.8K reputation)SSCommitted (1.8K reputation)SSCommitted (1.8K reputation)SSCommitted (1.8K reputation)

Group: General Forum Members
Points: 1752 Visits: 3427
Thank You Everyone
That is what I knew. I was just hoping that someone more intelligent that I would know of something else.

Thanks
Andrew SQLDBA
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum

































































































































































SQLServerCentral


Search