Maybe. Maybe not. Depends on the rows involved, the join type, other filters, complexity of the query, data types and a few other things. Add indexes, see if they help.

Temp tables are not necessarily memory resident, but anything that they query processor operates on will be in memory, temp or permanent, the query processor doesn't know what a disk is.

Ignoring joins, the permanent table needs a clustered index.