Because there is no data in the temp table when it gets created, there are no stats. Without stats, you get an estimate of 1. You can update the stats in the temp table prior to use. That may affect the plan generated.
"The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood"
- Theodore Roosevelt
Author of:
SQL Server Execution Plans
SQL Server Query Performance Tuning