Thanks everybody for your responses. I could not post the table structure etc because of policies. Here is what I found out after more testing.
From the execution plans, it looks like the SQL server has to use all the records from the tvp anyways as it does index seeks against the main physical table. That is why having an index on tvp does not matter.
There is one thing to notice here and that is how do you join the tvp against the physical table:
1. With the non-indexed tvp, when I use where exists() to filter records from the physical table, then the SQL Server does add Sort() in the execution plan after the Table scan of the tvp . If I convert it inner join then it removes that Sort().
2. With the indexed tvp, it does need that Sort() even if where exists() is used.
So when inner join is used, both non-indexed and indexed tvps perform same.
So I think the only advantage that an index gives is the uniqueness of the records coming in. The "Ordered" part of the index does not come in handy. I tried this out with the inserts as well and found out the same thing.
Let me know what do you guys thing.