I have transaction log backup failures randomly. SQL Server Log says 'Memory constraints resulted reduced backup/restore buffer sizes. Proceding with 6 buffers of size 64KB.', and the SQL Server Agent Error Log has quite a few errors like 'Unable to start Job Manager thread for job xxx', ' SQLServer Error: 768, Client unable to establish connection [SQLSTATE 08001]',  SQLServer Error: 768, SSL Provider: Not enough memory is available to complete this request [SQLSTATE 08001]'.
I am really low on the memory? It has total of 32 GB, but I set maximum to 20. Task manager shows it is using 99% of physical memory. Memoryclerk-sqlbufferpool has 32GB for the Virtual Memory Reserved, and 20 GB for the Virtual Memory Committed.
Can anyone advice? Thanks in advance.