I have configured a Listener (say AG_LITEN) with port 5152 using static IP on SQL 2012 named instances.

When i try to connect using Listener in SSMS it doesn't connect whereas when i pass the IP port number (which i have used to configure the Listener) along with Listener name (AG_LISTN,5152) it work like a charm. I tried doing manual fail over and the same connection string connect to the new primary node.

Give it a try.