We THINK we may have found the issue, we'll know within 24 hours. We noticed that the errors were occuring 2 minutes after the start and end of the "disk scrub" that the MSAs perform every day. This checks for disk defects and takes about 12 hours to complete. I'm guessing that this scrub somehow makes the controllers temporarily fail before recovering.

The latest firmware has a fix for this issue but, for now, I've disabled the scrub and we'll see if that helps.