It is a best practice to set a max server memory limit. Right now, you are starving the OS of memory and in the end you could be causing contention for memory.

You should reduce max memory for sql server. Here is an article with a guideline and formula.

http://www.sqlskills.com/blogs/jonathan/how-much-memory-does-my-sql-server-actually-need/