The SQLServer and SQLAgent login are both under login "local system" so I can't give those logins access to the backup server as far as I'm aware.

I'm able to backup my other databases to same location though so this shouldn't be the reason why or all my backups would fail.