• cphite (11/18/2015)


    Jeff Moden (11/18/2015)


    cphite (11/18/2015)


    jbrandenburg (11/17/2015)


    I have been working very closely with the storage admin. He has confirmed things appear to be setup properly. The SQL server and the backup server are on the fastest disks we have.

    If 150GB of backups is taking over a day, then things are absolutely not working properly. I would be very concerned about that level of performance because it suggests either a serious configuration problem, or a serious hardware problem. Which means that in addition to your backups taking so long to run, you also risk your backups not being accessible if the problem gets worse.

    That's probably the best post on this thread, yet. Even when I was backing up an uncompressed 360GB to a NAS, it wasn't taking even a half day. Something is really wrong and it just hasn't been found yet.

    I currently back up two 400GB each from two SQL servers to a NAS that is 12 miles away, over a dedicated data line, and it gets done in around 4-5 hours. We've gotten to around 3.5 hours if we increase the bandwidth of the data line, but that's too costly to maintain. The point is, we get around 85GB/hr mostly because of the data line; locally it's more like 150GB/hr.

    The OP is getting around 6GB/hr on what is, based on his description, essentially a local connection. Same SAN but different drives. This isn't simply bad; it's worrisome.

    For example, when we switched to a different type of storage, all of our backups were suddenly taking 4-6 times longer. Everyone said it's because of the "deduplication" being done on the new device and wanted to just live with it. Everyone except me and one of the NetOps guys. After some research and throughput testing, it did, in fact, turn out to be a "too long" cable that also had a kink in it. We replaced the cable with a proper length, unkinked cable and the new device starting doing the backups without any such delays and is actually 25%-50 faster (even with the deduplication and depending on the database) than the original devices that we were backing up to.

    Good point.

    The storage admin needs to check everything; drives, cables, the overall hardware. This isn't a case where you simply look at the main config page and say "Yeah, that looks about right..."

    This would be like if you took your car into a garage and told them you could only get it to 10MPH, and they give it a quick glance and tell you everything looks okay. No - something is wrong, and it's imperative you find out what. Even if it turns out to be something as simple as a bent cable or a config setting, you need to know.

    Not to pile on, but if the SQL servers and the backups are indeed on the same SAN, even on different disks, you have even more reason to get this resolved. If it is a serious hardware issue that inevitably takes down the SAN, you risk losing your data AND your backups.

    Our storage admin has begun with disk performance monitoring. As I do not want to throw this over the fence, I would still like to check any and all SQL Application related things that could be causing the delay. The one thing I did notice was that 1 database in particular is taking hours to complete. The database file (mdf) is around 177 GB. After the backup compression runs and the backup completes, the .bak file is down to 12 GB. Is that an expected amount of compression? The particular database is our ODS database for business intelligence. Any thoughts?