It is hard to tell what the issue is from just looking at the plan right here, but in the past when I have had issues with performance on linked servers (and much like yours here it was a linked server within a join to some other tables) the thing that has invariably helped the most for me was to bring the data from the Linked Server into a temp table or table variable in a pre-select and then join that table to the remainder, such that the final query is totally local.

IMO, the optimizer struggles when it comes to linked servers and by doing this you essentially help it out. It is worth a shot, I would say unless someone else has other options out there.