This is a blast from the past!
So let's look at this logically. WTH is a "primary key"? Quite simply it is a key (also known as a "candidate key") which is a column, or set of columns (note that they are not technically a "set", since order *does* matter at creation time, though you can use them out of order in join conditions and projections) that uniquely identifies rows in your table.
So what's the difference between a "primary key" and any other "candidate key" on your table? Here it is in a nutshell:
1) A primary key is always non-nullable; any other candidate key can be declared as nullable or non-nullable
2) A primary key is declared using the PRIMARY KEY clause; any other candidate key is declared using the UNIQUE constraint clause.
3) A table can have only one PRIMARY KEY; tables can have multiple candidate keys
4) A primary key, if declared on a table with no clustered index, will default to be your clustered index; if the table is already a clustered index the primary key will default to nonclustered. A unique constraint always defaults to nonclustered unless you specify the CLUSTERED keyword, and the table has no clustered index.
BTW, I realize that #4 is a LOT to remember, so your best bet is to ALWAYS specify the CLUSTERED or NONCLUSTERED keyword every time you create a primary key or unique constraint -- and you never have to worry about all those crazy rules. Another thing I would recommend is always naming your primary key and unique constraints. Give them short but meaningful names and it helps to follow a pattern. For primary keys, I usually go with PK_TableName, for instance.
Now, whether you define a key as a primary key or a non-nullable unique constraint just doesn't matter. Other than the items above, special handling by a few external third-party tools, and some people's odd misperceptions, it just doesn't matter. Here's what you need to know about the code you wrote above:
1) If you use an IDENTITY column, it is a "surrogate key". What this means is it is for internal use only. You should NEVER expose a surrogate key to an end user, nor to an external system. I have no doubt that there will be some people who read this and will want to argue the point, but I will make it really simple: the values in a surrogate key are subject to change on a whim. If your users depend on the identity value "1" to be "Canada" and you hardwire the value "2" for "Zimbabwe" into your client code, as soon as you rebuild your database later you will potentially throw all your dependent systems out of whack and piss off your users.
2) If you use an IDENTITY column do not rely on monotonically increasing numbers with no gaps. You can, and eventually will, have gaps in the numbers. IDENTITY columns do not guarantee there will be no gaps between 1 and 100. What they do guarantee is that every row will get a unique value that increases on each INSERT.
3) If you use a surrogate key, such as an IDENTITY column, you should always, always, always, have a candidate key declared with a unique constraint that defines the "business key". The business key is the set of columns containing business data (i.e., data that is relevant to your business users, systems and processes) that uniquely identifies each row. In your example above, the IDENTITY column you created is an internal surrogate key and the CountryName is the business key in this scenario.
So why do you even want a surrogate key? Simplicity and efficiency. The database engine can compare two integer values much faster than it can compare two strings (also I would suggest NVARCHAR instead of VARCHAR in your example). Multiply that by 1,000,000 comparisons in a join and you have a significant performance difference. It also tends to make queries much simpler to write when you join on a single column, as opposed to 4, 5 or 6+ columns.
Now, all that said, Countries are a HORRIBLE example for any of this. There is a freely available ISO country code list which provides full country names, unique two character and three character country codes *and* 3 digit numeric country codes. Every country has their own set of internationally recognized codes, including a numeric code, so a contrived IDENTITY column is completely unnecessary.
So at the end of the day, whether you want to declare a surrogate key as your primary key or just throw a unique constraint on it is a matter of 6 in one hand and half a dozen in the other.
A far more interesting and consequential conversation is where to put your clustered index.