I suspect you are using some process which is using nay extended procedures or clr ones causing memory leak

or

Have a external backup utility like veritas or litespeed purging memory

or

in any case set the max and min memory (remember both) to same value so that trimming is restricted completely for windows, and also enable lock pages in memory option in case its not already done