That's an SSMS error, not SQL Server. SSMS is a 32bit application, and thus is quite limited on how much memory it can address (can't recall if it's 512MB or 2GB that it "runs out" at). If you use too much, it'll return this result. It might not even be that particular query that caused SSMS to use a lot of your RAM, but something did. Closing and reopening SSMS (or opening a new one) will resolve the issue.
If you need to run a query that will return a HUGE result set, don't do it in SSMS.
Excuse my typos and sometimes awful grammar. My fingers work faster than my brain does :-P
Please always remember to encapsulate your code in IFCode Markup. For example [code=sql] [/code]
to read Jeffs Guide on how to post SQL questions, and get swift and helpful answers from the community