Please make sure if you have configured memory settings for SSRS in RSReportServer.config file.

For example(sample values only):

<MemorySafetyMargin>80</MemorySafetyMargin>

<MemoryThreshold>90</MemoryThreshold>

<WorkingSetMaximum>4000000</WorkingSetMaximum>

<WorkingSetMinimum>2400000</WorkingSetMinimum>

Refer to https://msdn.microsoft.com/en-us/library/ms159206.aspx