I had this issue when trying to drop the login of a person that had done an AOAG install\config. That login had granted connect permission to the HADR endpoint for the SQL service account.

Use this query to view endpoint information or just do * from sys.server_permissions to see everything.

SELECT EPS.name, SPS.STATE,

CONVERT(nvarchar(38),

SUSER_NAME(SPS.grantor_principal_id))AS [GRANTED BY],

SPS.TYPE AS PERMISSION,

CONVERT(nvarchar(46),SUSER_NAME(SPS.grantee_principal_id))AS [GRANTED TO]

FROM sys.server_permissions SPS , sys.endpoints EPS

WHERE SPS.major_id = EPS.endpoint_id

ORDER BY Permission,[GRANTED BY], [GRANTED TO]

Used these queries to change the grantor and also re-grant connect on the endpoint for our SQL service account.

ALTER AUTHORIZATION ON ENDPOINT::Name_of_endpoint TO sa;

GRANT CONNECT ON ENDPOINT::Name_of_endpoint TO [SQL_Service_Login];