First, you trigger must be INSTEAD OF, not FOR.
FOR triggers are executed after insert is done, so, it there is a constraint - it will raise the error before the trigger is executed.
Second, if I understand you explanation right, you want to avoid duplicate names where they are NOT NULL, and if inserted name is NULL then exclude duplicate CompanyID - is it right?
If yes, then your filter must look like this:
FROM [dbo].[Company] C
INNER JOIN inseted i ON i.[CompanyName] = C.CompanyName
OR (i.[CompanyName] IS NULL AND C.CompanyName IS NULL AND i.CompanyID = C.CompanyID)
Not sure how you want to deal with cases when 2 records have the same CompanyID, but 1 of them has CompanyName NULL. To me - it's duplicate as well, but it's OK according to your definitions above.