So, the first thing is that in this situation you absolutely want to set the Max Memory for the database engine, I can't recommend a value because I don't know enough about the server. Here is are some questions that need to be answered:

1. Is the SQL Server just hosting SSRS and the Report Server databases or does it host other user databases?

2. Please define "high memory usage" and problems that this is causing?

3. Are there any jobs running during this time frame (index maintenance, data loads)?

4 What is the main workload on the server, reporting, OLTP?