SQL Server will always take as much memory as you allow it to take.
It uses it to cache execution plans, returned resultsets, derived tables, etc.
It does not erase it until it runs out of memory.
If you want to limit amount of memory available to SQL Server set the limit explicitly.