Use a traditional upsert.
What you're trying to do in your original MERGE is to update rows which don't exist.
Perform your inserts first from a deduped source set - that is, deduped on whatever you are using to determine uniqueness. The remaining rows of your source set are all updates. You could use ROW_NUMBER() to distinguish between the two. EXISTS is the first option I'd experiment with for determining which of the deduped rows to insert into the target.
Edit: clarity
For fast, accurate and documented assistance in answering your questions, please read this article.
Understanding and using APPLY, (I) and (II) Paul White
Hidden RBAR: Triangular Joins / The "Numbers" or "Tally" Table: What it is and how it replaces a loop Jeff Moden