Dear Arooj,

Do you have any scheduled reports running on the server? Scheduled reports takes up a lot of memory, setting up the WorkingSetMaximum,WorkingSetMinimum value in RsReportServer.config file will help setting a CAP on the memory utilization by SSRS.

You can start with that and check the performance.

You can check the listed article for help on setting up these parameters.

# http://www.toadworld.com/platforms/sql-server/b/weblog/archive/2013/07/09/configuring-available-memory-for-sql-server-reporting-services.aspx

Hope it Helps..!!