Get a copy of sp_whoisactive (amazing freebie from Adam Machanic found on sqlblog.com) and check for tempdb usage, IOs, query plan, etc. Parameter sniffing could be at fault here since you mention it runs fast directly from SSMS. PFS/SGAM latching could be to blame if you have tempdb object creation.

Honestly there are 1001 things that could be at fault here.