Doesn't sound like a memory leak. You're over-working SQL Agent running linked server queries (which do not use the buffer pool) and it's requiring 3 GB of memory. I would not expect SQL Agent to release memory it has taken even if it's not currently all in use.