Here are some of the things I'd look at.

What do your typical performance and counters look like?

What's the target server memory, and total server memory under the sql memory manager counter?

What's the page life expectancy look like when this process is running and not running? What's the pagefile usage look like?

Is there anything in the error logs? Do you have an exceptional amount of faults in the memory counters that may indicate a bad (or several) sticks of ram?

Have you benchmarked the ram, and veified they are the exact same model, speed, size and timings? Is it the correct ram for the motherboard?

What does the cpu look like when the process is happening? What's the cpu temp look like?

What are your wait types looking like?

Does the operation you're running have any type of query hints or plan guides that may be effecting it? What's the nature of the CLR? Are you sure the CLR doesn't have a memory leak?

Have you tested it on a completely separate machine?