I can't check this right now but I'll will ... there is a question then ... assuming you're right - why during failback (Primary becomes again Principal and secondary Mirror) everything works without restarting the application? Also why this works when the secondary reconfigured to listen on default port 1433?
It really seems that the main issue here is the non-default port rather than anything else.