If I understand correctly the procedure is called multiple times until all rows are processed. Every time its called it needs to join with the large remote data set. I would think the large remote data set is being dragged over the network every time.

Can you pull the remote data set back to the local server into a temporary or permanent table? If the data is staged locally then your joins will be much faster.

My other thought is to call the procedure once and process all the data at one time. Is the @rowsToProcess parameter an attempt to address performance issues?