sysadmins can connect to any database/AG via any listener. Accounts with more restricted permissions cannot. Therefore I'd always set up one listener for each AG. If you don't, and the AGs fail over at different times, you can get some weird things happening.