I'm getting this same error on a 64-bit workstation w/ 24GB of ram, 15GB that Win7 lists as "available". This is unlikely to be a windows limit and more likely a leak or bug within SSMS.

I was just doing a TOP 100 query against a few small tables on different servers. They were all behaving the same way. Another instances of SSMS that I had open was behaving fine.

In my case, it's SQL Server 2012 SP1 (11.0.3128.0)

aka: the most problematic release of SSMS to date!