When you suspect high memory usage by non-bufferpool components, use this query to find out large consumer.

select Type, SUM(Multi_pages_kb)/(1024.00) Multi_MB From sys.dm_os_memory_clerks

Group by type order by 2 desc