Okay, understand your point that it appears to be a sql server unloading dlls. However, the issue does appear to be fairly common with Oracle 10 client. So my thoughts are to address Oracle Client issues first.

In searching further I found this these references http://sqlblogcasts.com/blogs/christian/archive/2008/01/07/sql-server-memtoleave-vas-and-64-bit.aspx and http://www.johnsansom.com/sql-server-memory-configuration-determining-memtoleave-settings/ to monitor and diagnose Virtual Address Space.

Hope this helps. . .