You can reference the named instance in the linked server configuration
On the General page, in the Linked server box, type the name of the instance of SQL Server that you area linking to.
Note
If the instance of SQL Server is the default instance, enter the name of the computer that hosts the instance of SQL Server. If the SQL Server is a named instance, enter the name of the computer and the name of the instance, such as Accounting\SQLExpress.
http://msdn.microsoft.com/en-us/library/ff772782%28v=sql.105%29.aspx
But it does sound like some sort of port opening issue between the two servers.