I don't believe the AWE option makes any difference on a 64-bit system so that shouldn't be an issue.

See these two articles for more explanation:

http://blogs.msdn.com/b/psssql/archive/2009/09/11/fun-with-locked-pages-awe-task-manager-and-the-working-set.aspx

http://www.sqlskills.com/blogs/paul/a-sql-server-dba-myth-a-day-530-awe-must-be-enabled-on-64-bit-servers/

Since it's a VMware server, have you checked for memory ballooning? And do you have a memory reservation configured for your VM? Also, is there memory pressure on the physical host system? I would begin to look at these things if you still see high memory allocation even after reducing max server memory.