It sounds to me that the memory is set to default on the database engine and in turn SQL will take all the memory on the machine if not set under the SQL engine.

If the machine has 64GB give 56 to 60 to SQL and 4 to 8 for the OS.

Do you know where to set the memory on SQL?