One way performance could be affected is if you use temporary tables. You may have a database which has a stored procedure which uses a temporary table.
I think there are other ways that a database will use the tempdb, but I don't know them offhand.
Your right Temp Tables as well as Table Variables, SQL Internal Work tables, the SQL version store, re-indexing jobs that specify sort in tempdb, and many other things are done within the Tempdb for not just TDE databases but for all databases, as well as for SQL internal processes.
Just to repeate though TDE uses the Windows API to encrypt the files on disk. Pages in memory are NOT encrypted, the TempDB files in memory would feel no ill effect from this if they have enough memory to opperate in.
This would effect a server during the checkpoint where all pages are flushed to the disk. Which once again depending on how the server is sized, the load in place, and the volume of users may make a little, a lot, or no difference whatsoever.
Hopefully this will get expanded by Microsoft or on some SQL blogs some where, as there is not a lot of documentation out there on this overall effect right now.
But you were right on in the suggestion that Temp tables could be effected but they would need to be so large that they spill onto the disk as if they are in the memory they should be unaffected.