Do you have any scheduled reports running on the server? Scheduled reports takes up a lot of memory, setting up the WorkingSetMaximum,WorkingSetMinimum value in RsReportServer.config file will help setting a CAP on the memory utilization by SSRS.
You can start with that and check the performance.
You can check the listed article for help on setting up these parameters.
Hope it Helps..!!
And Your Lord Never Forgets...!! (64:19 -- Quran)