The sql server agent job runs under the credentials of the service account.

Question 1 - Do you have the same performance issue when logging into SSMS using that account?

The min / max memory settings are sometimes forgotten about when building out a new server.

Question 2 - Did you use the correct settings?

http://www.sqlservercentral.com/blogs/glennberry/2009/10/29/suggested-max-memory-settings-for-sql-server-2005_2F00_2008/

Question 3 - Is this a physical or virtual server that you have SQL Server installed on? If it is a virtual server, there can be issues with the ballon memory manager, I am not a VM expert.

Question 4 - Make sure you have the latest service pack and cummulative updates for SQL Server 2012.

Please answer these questions. It will give me a better idea of what you are dealing with.