If you have reasoned out the issue to be Memory, does the simple query do lot of in memory sort operations. May be creating an ordered/grouped by covered index can help. Check the plan for each query and get an estimated size of data. Also, you mentioned that the OS is demanding more memory, do you see lot of paging?