Have you looked at your wait stats? Although you haven't provided any detail I can guess that such high latency could be caused by too much IO read: not enough spindles and/or datafiles in the database for the amount of requests that need ot be serviced.
How is your TempDB set up?
Heh... or some seriously performance challenged code with non-SARGable predicates and insufficient criteria that cause accidental many-to-many joins.
We had some weekly issue with tempdb bottleneck using an SSD drive with 4 files and a log file. We split the SSD mirror into two and place two tempdb files on each drive, and moved tempdb log file to a 3rd drive. Problem no more.