I should have added the kicker I learned from HP support. There need to be two batch files to start and stop the QuestLauncher (QAM Launcher) service when the instance starts or stops. The Quest manual mentions it, but I had to use "C:\WINDOWS\system32et.exe start QuestLauncher3566" instead of just "net start QuestLauncher3566" like the manual says. Maybe that is due to the 64-bit Windows server environment.

(I'm struggling with the text editor here. I just can't make my correction stick. Back slash 'n' must be a special code. The path should have "system32 [back slash] net.exe")

And yes, every instance has to have its own port number in spite of documents one can find on the HP web site that say multiple instances can use the same port. Maybe the fact that PS is sort of a cluster makes the difference.