Yes you are right. For Optimizer TVF is like black box.

Simple Single statement TVF can be in lined by SQL but for multi line TVF SQL Server will repeatedly

execute the function – for every row in the result set. Which will cause the CPU to spike a lot.

Have you tried replacing the TVF with inline SQL query logic?

Also one of the issues with functions that they don't get good stats from optimizer and sql server has a no way to know at compile time how much data it is going to return and it works on assumption that only one row is going to return from function.

Have you tried seeing the execution plan just for TVF for different kind of parameter to see if that will help?