One reason could be that you're using UNION instead of UNION ALL.

When combining two queries using UNION you'd force an (internal) DISTINCT to be applied to the result set, usually taking quite some time.

When using UNION ALL you'd get the results of each query, including dups.

It depends on the statements you use: If you can guarantee that there will be no duplicates by the nature of the query or if you can accept dups then use UNION ALL instead of UNION.

You can verify how much the sort operation will consume you could compare the actual execution plan of each statement.