Assuming your indexes are appropriate for your query;
Create a view to pre-filter your data on the remote server. Ideally, create a single view that has all of the joins and data columns you need from the remote server. That way the remote server will take some of the load of filtering and joining. Use the view in your query to only retrive the data that you need from the remote server.
In a linked server join, the host server retrieves ALL of the rows from the source tables before attempting the join. It then performs the joins and filters locally.