Wow - now I am down to 1 sec. after I added the index:

CREATE NONCLUSTERED INDEX [IX_BRWEEKHIS_TEST]

ON [dbo].[BRFCLSeaWeekHis] ([PortFrom],[PortTo],[QuoteType],[Year])

INCLUDE ([CurrencyCode],[RateTotal],[SeaTotal],[Total])

However, the last two conversions I don't understand, because they seem correct when declared into the temp table. I will dig a bit more into that, but now it is really great! Thank you very much!