I don't believe the AWE option makes any difference on a 64-bit system so that shouldn't be an issue.
See these two articles for more explanation:
Since it's a VMware server, have you checked for memory ballooning? And do you have a memory reservation configured for your VM? Also, is there memory pressure on the physical host system? I would begin to look at these things if you still see high memory allocation even after reducing max server memory.