I've been using this architecture on Win Server 2k3 32bit and every windows os I've had before that with no problems. I'm not sure whether it's a Win Server 2k8 thing or a 64bit thing. However, it does not seem to be a 64bit thing because it works perfectly when both instances are run exclusively INSIDE and exclusively OUTSIDE of SQL Server on the 64bit setup.
The main difference between Mutexes and Semaphores is that Mutexes are controlled by the operating system so I think this is platform issue which I think it can be fixed if you open your VB.NET code and recompile it for x64. The reason is you said if run separately it works so it is related to the use of two x64 system that depends on named thread synchronization.
Kind regards,
Gift Peddie