Thinking about it more, you should be able to get reasonable performance, even pivoting the data, assuming the indexes are clean, especially the clustered indexes. How did you set those up?
"The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood"
- Theodore Roosevelt
Author of:
SQL Server Execution Plans
SQL Server Query Performance Tuning