CROSS APPLY is a row level operator. So your table_valued_function will be run once for each row the rest of the query returns.
So my guess is that the query:

Your jpg file attachment is showing a later date for CF, is this what you meant. I've assumed you didn't?
Here are two different ways to get the results:

This is an example where De morgan's law can be applied.
(not A) AND (not B) = not (A OR B)
So the expression 
...

If you have performance problems with a query like this it's probably due to missing indexes. Do you have indexes on the tables?
Have you tried executing a query and...

If you run the stored procedure with SET STATISTICS TIME ON you will be able to work out which statement(s) is/are taking up the time.

;WITH CTE AS
(
    SELECT UserKey,
           COUNT(*) Count
      FROM UserTable
     GROUP BY UserKey
)

I would also take the elapsed times of the query.
What is the query you were running?