(BALLPARK Numbers. No two sql server instances have the same workload.)
Dedicated to 1 instance of SQL Server (leaving 1.5-4 GB for the OS):
GB RAM/Max Memory Setting
If you've got more stuff running (multiple instances, SSAS, SSRS, SSIS, etc) then you'll need to figure out what makes sense for your situation.
For example, If you've got a server with 32 GB, running SQL, SSRS, SSAS and SSIS.
After watching memory usage, you determine that the SSIS packages max out around 800 MB total at any point in time. SSRS seems to stay around 1.2 GB.
So, do some math, 32 Gb - 1.5GB (for the OS) -.8GB (SSIS) - 1.2GB (SSRS) = 28.5GB for SQL and SSAS to split.
You experiment and determine that SQL getting 18 GB and SSAS getting 10 is a good split and those settings become your max memory settings. You may also decide to set min memory in case something bad happens (rogue SSRS report), so you decide that sql should always have at least 8 GB, and SSAS at least 6.
You've covered your bases, and you've got 14GB of wiggle room if something goes haywire, but on a good day, most of your ram will be utilized without paging.