In logical terms there is absolutely no difference. A key (i.e. a candidate key) is a set of attributes which is required to be minimally unique and does not permit nulls. A table may have multiple keys. By convention one key per table is designated as "primary" - usually a key that has some special significance for the designer or user of the database, e.g. a key used as a foreign key in other tables. That's only a convention however and is or essentially should be just a label of convenience.

In implementation terms there is mostly no difference either. A primary key may be clustered or non-clustered and most of the features supported by SQL Server can apply equally well to any unique constraint or unique index. Unfortunately the SQL syntax designated PRIMARY KEY is actually required in a few places or makes a difference to how some features operate. Replication is one example. Also the PRIMARY KEY syntax is relied upon by some database design and development tools to drive certain features. These limitations of SQL Server and other software are unfortunate because they actually work against the original concept of a primary key - no longer can you necessarily choose the "preferred" or "most significant" identifier as a primary key because you also have to consider the possibly unintended consequences for other features or software.

In practice therefore there is no easier answer to your question. For the most part it makes no difference at all and that's the way it should be. Any actual difference depends on how the key will be used. It also depends on what your motivation is for defining such a key in the first place - what difference does it make to you? If you can't answer that question then maybe you have no justification for defining it as "primary" in the first place.

Here's a fun example - hypothetical but a perfectly realistic example of multiple keys. Imagine a table of marriages (for simplicity I'm only considering monogamous, male-female marriages). There are two columns, Husband and Wife. Both columns are keys because we don't want to allow polygamy. There are also referential integrity constraints to ensure that only women can appear in the Wife column and only men in the Husband column. Now, which is the "primary" key - Husband or Wife? Does it make any difference?