Backup issue, may be a problem with 3rd party backup app

  • All,

    I've got an interesting backup issue on a production server, that has me and the backup admin getting ready to tear our hair out. I've got one database on this server, that when our third party backup application goes to back it up, takes HOURS to backup. Some info, first on the server and db:

    Windows 2008 R2 (current SP) / SQL 2008 R2 SP3

    It is a virtual machine on VMWare, underlying storage for the VMWare is on a SAN

    3rd party backup software: Commvault

    There are about 29 databases on this server, only this one is a problem

    The DB in question has a 150GB MDF (~40GB used) and a 205GB(!!!) LDF (currently using ~20GB) and is the largest DB on the server

    ALL the databases on this server are encrypted with TDE

    All the DBs live on the same drives, which are not the OS drive, data files and log files are on separate drives

    When the backup application tries to backup a database, I can see a normal "backup database..." command in the running queries. It does backup to a "virtual device" (two actually)

    Some information I've pulled out show that the two big wait types when this DB is being backed up are ASYNC_IO_COMPLETION and BACKUPIO (which would tend to point to a config issue with the backup application) The currently running differential backup shows a total elapsed time of 183.5 minutes. If I run a manual backup of this database to disk (full backup) it takes ~20 minutes. This is the ONLY database on this server having an issue being backed up.

    I took a look at the performance counters for backup / restore throughput (as well as disk read / write time for the data and log drives) and all looks OK. Watching the throughput counters for both the problem DB and a log backup running for another DB, they looked similar (same scale) As a point in fact, the average for the problem DB is about 10x higher than the other DB.

    Any suggestions on anything else as far as SQL goes that I could look at?

    At this point, I'm even considering setting up a maintenance window with the customer, taking a backup of the DB to disk, detaching the DB, renaming the files, restoring the backup, then deleting the old files on the odd chance it's something weird.

    I do run weekly DBCC CheckDB using SpaghettiDBAs script, and it continues to come back clean every week (the most recent being the 19th)

    Thanks all,

    Jason

Viewing 0 posts

You must be logged in to reply to this topic. Login to reply