Nothing is really standing out as holding onto a large amount of virtual memory at the moment. You can also have this issue if the VAS becomes fragmented and there no way to defragment it short of rebooting. What are you using to do backups? Are you running any third-party products or extended stored procedures on the server?

The best move really would be to move onto a 64-bit platform.