If you've set max server memory to 20GB, then SQL will increase it's memory usage up to 20GB and it will stay there. It will not deallocate memory unless the OS is under memory pressure. Again, this is normal, documented, expected behaviour.
Gail Shaw
Microsoft Certified Master: SQL Server, MVP, M.Sc (Comp Sci)
SQL In The Wild: Discussions on DB performance with occasional diversions into recoverability