You set the max memory so that the buffer pool doesnt consume all your servers resources resulting in SQL starving the OS of resources and the time it takes for SQL to release the resources back to the OS which would involve a potential large amount of paging depending how much RAM SQL was force to release back to the OS, resulting in detremental performance while the operations are performed.

Objects such as CLR, linked servers, COM objects all use memory from outside of the buffer pool so they will increase your memory usage.