I'm not sure how true that is... what about a self-join?
Surely the speed of the joins is dependent on the columns being joined (i.e. key columns vs non key columns), whether those columns are indexed, up-to-date statistics, and a tiny little bit of data type (e.g. int or varchar).
I wouldn't say row count is an indicator for performance