It's definitely permissions. AFAIK, however, it's the SQL Service account, not the agent account, that needs permissions.

The agent logs in just like you do from your workstation. The backup is run by the SQL engine, and it's service account needs permission.