Memory Usage

• Dan Ellison

  Valued Member

  Points: 52

  More actions

  February 11, 2009 at 9:42 am

  #126374

  We are constantly failing due to too much memory consumption. I have been playing with DefaultBufferMaxRows and DefaultBufferSize for some time and having trouble corrolating the settings to actual usage or the documenation.

  The first problem I have is with a Merge Join. Both inputs should be providing the same number of rows with a 1 to 1 relationship. With MaxBuffersPerInput=5 on the Merge Join component, and DefaultBufferMaxRows=10000 on the Data Flow, I would expect neither input to get more than 50,000 rows ahead of the other. However, in reality, when one input is finally exhausted, it is 3.9M rows ahead of the other and approximately 200-250MB of memory has been consumed.

  My second issues is with log events such as:

  BufferSizeTuning J00000WITD010 COURTS\ellisond Load Case_Event {a922a9b2-d278-4686-8017-cfe114f02859} {ADD4C059-C45F-41A5-BC1B-04738A550E01} Rows in buffer type 21 would cause a buffer size less than allocation minimum, which is 65536 bytes. There will be 1638 rows in buffers of this type. 2/10/2009 1:49:19 PM 2/10/2009 1:49:19 PM

  This event leads me to believe that DefaultBufferMaxRows was set to something less than 1638 rows and for this particular buffer, it was raised to 1638. However, I know that DefaultBufferMaxRows is at the default of 10,000.

  Am I reading these things incorrectly?

  Thanks,

  Dan