Are the SID'd for the application user on primary & mirror server are same and also default DB for that user is set?
I would suggest to test this via failing over to mirror and try to connect via application user credentials?
Please see the below thread:-
http://www.sqlservercentral.com/Forums/Topic1451617-1549-2.aspx