Nested loop join on the slow plan between many to many and on the fast one it is nested loop 1 to many.

So the faster operation should be Many to 1 times faster.