But how many rows are being returned and it sounds like the aggregation is being performed in the table with grouping, so that will add some overhead. Look at performing the aggregation ahead of time if you don't need the detail.

You are running SQL Server, IIS, Report Server, ... and what else on the server. It is just not Report Server. Monitor the resources on the server and run a Profiler trace on SQL Server to see what is going on. Because of the volume of data you might need to look at making some modifications to the packet size possibly in IIS or test running the report with a smaller dataset.