Wow - now I am down to 1 sec. after I added the index:
CREATE NONCLUSTERED INDEX [IX_BRWEEKHIS_TEST]
ON [dbo].[BRFCLSeaWeekHis] ([PortFrom],[PortTo],[QuoteType],[Year])
INCLUDE ([CurrencyCode],[RateTotal],[SeaTotal],[Total])
However, the last two conversions I don't understand, because they seem correct when declared into the temp table. I will dig a bit more into that, but now it is really great! Thank you very much!