A good tip for dealing with instances is to use a variation of the standard sp_addlinkedserver syntax and give the linked server a different name.

EG: sp_addlinkedserver 'Server_Inst', ' ', 'SQLOLEDB', 'Server\Inst'

Removes the need to worry about the square brackets