I'm getting this same error on a 64-bit workstation w/ 24GB of ram, 15GB that Win7 lists as "available". This is unlikely to be a windows limit and more likely a leak or bug within SSMS.
I was just doing a TOP 100 query against a few small tables on different servers. They were all behaving the same way. Another instances of SSMS that I had open was behaving fine.
In my case, it's SQL Server 2012 SP1 (11.0.3128.0)
aka: the most problematic release of SSMS to date!