How big is the underlying data set?

If you have an order by it has to return all the data, sort it and then look at the top 100,000, where as if you remove the order by it just gives you the first 100,000 rows that it comes across.