I've never had a performance problem because of such switches but if I were forced at gunpoint to make a conjecture, I would unfortunately still have to say I don't know because I've never tested for that.
I can, however, tell you that SQL Server will prevent the drop/rebuild of the synonym if someone is still using it. THAT I've tested for! It will sit and patiently wait to get exclusive use.
--Jeff Moden
Change is inevitable... Change for the better is not.