Correct. The availability group is at the database level. Server-level permissions are kept in master and that's not going to be part of your AG.
Just don't change any of the database user's permissions on the primary as those will be transferred.
p.s. The 'With SID' portion is critically important. If you don't, the user will be orphaned on the secondary and the person won't be able to get access. So you need to CREATE LOGIN ... SID = ... on the secondary, explicitly specifying the SID that the login has on the primary. This is for SQL logins. For Windows logins, the SID comes from AD and hence will be the same on both servers automatically.
Microsoft Certified Master: SQL Server, MVP, M.Sc (Comp Sci)
SQL In The Wild: Discussions on DB performance with occasional diversions into recoverability
We walk in the dark places no others will enter
We stand on the bridge and no one may pass