The 'optimize for adhoc workloads' sp_configure option only affects the adhoc cache, so all of the other caches will still keep the full plan for single use executions, which is where more of the memory is going, not directly to the adhoc cache. There is no option for minimizing caching of single use procs or prepared plans, so this may just be the nature of things for your workload.