Once SQL Server has grabbed the memory, which it will do for caching, it tends to hold onto it unless it's needed by other applications. And then it grudgingly lets go.

Is this 32 or 64 bit server? Is it affecting performance? If not, this is normal, and leave it alone.