So it seems this is definitely an issue of parameter sniffing.

While option(recompile) did nothing, option(optimize for unknown) brought the dynamic query down to 3 seconds. Still slower, but I can live with this.

Is there some more correct way to fix this, or would simply leaving that as part of the query be the best solution?