I just cleared the wait stats on the test box and re-ran the query to see if it helped track down the cause.

When I run the query directly on the server (Taking around 8-9 secs currently)

I get the wait types:

ASYNC_NETWRK_IO (50%)

PREEMPTIVE_OS_WAITFORSINGLEOBJECT (50%)

However, when I re-clear the wait counters, and run the same query from my desktop, I DON'T get the PREEMPTIVE_OS_WAITFORSINGLEOBJECT wait type, and the counters produce:

ASYNC_NETWORK_IO (99.9%)

Not sure why this is, when querying the same server, and I haven't taken wait times from the prod box to compare against yet...