You might be running out of restore threads on the mirror and need to enable trace flag 1485. The following is an excerpt from a SAP on SQL whitepaper (page 87) which can be found here:
•Database mirroring uses parallel threads to restore the transaction log records that were shipped to the mirror instance. By default it uses one restore thread for every 4 CPU threads. In our large SAP ERP encryption example, this still amounted to 16 threads encrypting on the principal, but only 2 threads were restoring on the mirror side.
If your system encounters an imbalance like the one we encountered, you should consider using trace flag 1485 on the mirror instance. This trace flag specifies one restore thread for every CPU thread. In our large SAP example, a 1:1 ratio between encrypting threads on the principal and restoring thread on the mirror could have been achieved with this trace flag.