I would expect that to be network related or memory related (experts correct me if I am way out to left field here).
If the network connection on the server is a Gb connection (1000 Mbps), the report is 166 MB. Different units, but lets pretend they are the same for simplicity sake (MB and Mb are not the same thing). that report is using 10% of the bandwidth to send the data to the end user. Plus using 10% of the bandwidth to pull the data to the server prior to processing it and sending it to the end user. Now, lets say you have 100 users on the server all pulling data at the same time, with 10 of them being users pulling that specific report. That will fill up the network bandwidth pretty quickly.
Rendering it locally (within the application) uses half the bandwidth as you pull data from SQL to local and then do all the other processing local.
Now, before jumping on the "network is slow" problem, I would investigate it and determine if it is the cause. Check perfmon/task manager on the server while you are loading the report. Does the network get saturated? If it does, you need to reduce the data set for the report or learn to live with it being slow. If it isn't saturated, check out the CPU, disk, memory, etc.
It could also be that the application server has more memory than the SSRS server and the SSRS server can't handle the load from that report and it is paging to disk.
I would also look to see if you can improve that SQL query. 10-20 seconds sounds like a long time to get the data, especially for only 100,000 rows. Now, 100,000 rows at 166MB 10-20 seconds doesn't sound horrible, but also sounds like it MAY have room for improvement.
I would first try to decide which problem you want to solve first. SSRS being slow, SQL query taking 10-20 seconds, 166 MB of data being pulled for the report. To me it doesn't sound like parameter sniffing UNLESS you are calling a stored procedure for the data AND you are using different parameters on SSRS and from the application/SSMS. If the parameters are the same, then it is (likely) not parameter sniffing.
As another thought, it could also be blocking. If it is sometimes slow from SSRS and sometimes quick, I would guess that blocking is a bigger problem than anything else. If it is consistently slow in SSRS, then it is sounding like either too much data or some resource constraints.
Just my 2 cents.