Have you tried applying a server side trace to see what is happening with the Database during the backup period, and maybe running perfmon counters to check throughput of the data across the network.

Might also be worth checking the availability of the Tape device during this time to make sure it is not cycling the tape. Possibly try running a ping every 2-3 minutes between the server and the tape device to make sure times don't suddenly dropoff during your backup.

The only real option is to look at each part of the process and apply relevant logging to it until you find out where the problem exists then deal with it.