All your guesses are true but my issues is not mainly about the performance but the weird behavior that is being demonstrated by SQL.
I am using 3 table variables and i get the data into the first 2 and then use them to populate the 3rd one.
So just changing the 3rd table variable into temp table to which i am just inserting this data from the same query is boosting the performance.
And i can't believe that this is because of any memory issues because i have tried this same query in 3 different servers all having different configuration.