Commvault thinks log chain broken, launches full backup during business day

  • Anybody else had this issue with Commvault for sql backups? Nobody is running sql backups outside of commvault. The sql log shows successful log backups every 15 minutes as expected, but somehow Commvault gets "confused", possibly due to a very busy sql server, and launches a full backup. Not only could this impact things during the business day, but it left us with no log backups for the 14 hours it took for this full to complete.

  • I haven't seen this reported, but I'd start complaining to Commvault.

    I would also not configure the software to run a full if the log chain is broken. You might do this for some specific reason and kicking off a full you don't need can be an issue.

  • Yes, I'm encouraging our Systems Team to open a ticket with Commvault. I don't have access to it but it seems to be designed for one thing: maintain viable backups no matter what. It does it's own thing if you switch recovery models and apparently is prone to launching a full when it "decides" that the log chain is broken.

  • Hello

    Was a support call ever raised with CommVault? If so, what was the result?

    We're experiencing a similar issue.

    Many Thanks

  • We have CommVault 8 and yes we experience the exact same thing from time to time...our system is very busy and I have never gotten an answer from CommVault or our Infrastructure guys as to why it happens. We don't have full access to investigate all the log files and in previous attempts to upload the logs when this occurs to CommVault fell on deaf ears.

    In our environment we have terabyte databases that this occurs to and of course it is in the middle of the day...which obviously impacts performance when a full kicks off at 8 a.m. on a Monday.

    Our work around is as follows which is the only way we could get around it outside of disabling CommVault until off hours to re-run a full backup.

    If you notice an event like you listed above and see a full backup running....you can kill the full and then run a differential backup to reset the log chain.

    It is the lesser of two evils and has worked for us to get the log backups working again.

    BTW I got that idea from Paul Randal's blog entry....THANKS Paul!

  • Disclaimer: I have no experience using CommVault, and the following is only based on my experience working on SQL Backup for the last 7 years.

    Is it possible that CommVault is querying the backup history tables to determine if a full backup is required, and the query is timing out due to the busy nature of the servers, and hence decides to run the full backup? The backup history tables can get very large if they are not regularly purged (e.g. using sp_delete_backuphistory), and depending on the query used by CommVault, may be slow because there aren't that many indexes created by default on those tables.

    If the above is true, perhaps there is a setting within CommVault to increase the query timeout value? Otherwise, I would suggest purging old data from the backup history tables if possible, and/or create additional index(es) on the table(s) to help speed up the queries used by CommVault. It isn't good practice to create indexes on system tables, but you don't have much of a choice here.

    Hope this helps.

    SQL BAK Explorer - read SQL Server backup file details without SQL Server.
    Supports backup files created with SQL Server 2005 up to SQL Server 2017.

  • We're in the process of moving to Commvault, and if i read the documentation correctly, it defaults to a full backup after 24 log backups. There is a registry setting that can change this value.

  • For us Commvault runs a log backup every 15 minutes, plus fulls on Sunday night and differentials once each weeknight plus Saturday. It's hard to imagine having to change a registry setting to increase a default of 24 log backups, but since I don't have access to the Commvault GUI yet, who knows.

    Backups are important, but so is performance, so having a product incorrectly launch a full backup during the business day is inexcusable.

    Fortunately I think we're going to be using snapmanager to get copies of production to a secondary system, and just use commvault to backup the secondary.

  • I managed to find an option for this by enabling the 'Disable Log Consistency Check' checkbox in the SQLSettings for each sub client.

    https://documentation.commvault.com/2022e/expert/61539_properties_of_sql_server_sql_settings.html

    Note : The risk involved with this option needs to be analyzed based on your environment.

  • vinchackochandy wrote:

    Any fix regarding this thread?

    We are frequently facing this issue.

    Considering this is a 12 year old thread, I'd suggest either starting a new one as things could easily have changed in the last decade, or (like Steve recommended back in 2011) reaching out to Commvault's support team.

    Thom~

    Excuse my typos and sometimes awful grammar. My fingers work faster than my brain does.
    Larnu.uk

  • As a bit of a sidebar, have you checked for code that either takes a backup somewhere outside of CommVault and have you checked to make sure no one wrote some code to "temporarily" change the Recovery Model to something other than FULL in order to try to take advantage of "Minimal Logging"?

    --Jeff Moden


    RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
    First step towards the paradigm shift of writing Set Based code:
    ________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

    Change is inevitable... Change for the better is not.


    Helpful Links:
    How to post code problems
    How to Post Performance Problems
    Create a Tally Function (fnTally)

  • No such actions were performed.

    1. No backups running outside the software.
    2. No log chain breaks.
    3. No recovery model change
    4. No point-in-time restore / changes in the system database.
    5. No missing in the backup history
  • I managed to fix this by enabling the option 'Disable Log Consistency Check' checkbox in the SQLSettings for each sub client.

    https://documentation.commvault.com/2022e/expert/61539_properties_of_sql_server_sql_settings.html

    Note : The risk involved with this option needs to be analyzed based on your environment.

  • Yowch.  That doesn't seem right.  Thank you for the feedback.

    --Jeff Moden


    RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
    First step towards the paradigm shift of writing Set Based code:
    ________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

    Change is inevitable... Change for the better is not.


    Helpful Links:
    How to post code problems
    How to Post Performance Problems
    Create a Tally Function (fnTally)

  • vinchackochandy wrote:

    No such actions were performed.

    1. No backups running outside the software.

    I presume you have double checked this by looking at the tables in MSDB:

    SELECT S.backup_start_date, S.backup_finish_date, M.physical_device_name, S.[database_name], S.[type], S.is_copy_only
    FROM msdb.dbo.backupset S
    JOIN msdb.dbo.backupmediafamily M
    ON S.media_set_id = M.media_set_id
    WHERE S.backup_finish_date > '20230314'
    ORDER BY S.[database_name], S.backup_finish_date DESC;

Viewing 15 posts - 1 through 14 (of 14 total)

You must be logged in to reply to this topic. Login to reply