Why? That's the puzzle. It looks as though the optimizer is not really building an intermediate resultset of numeric values, but rather is applying the various conditions (col>=1, col<=6, IsNumeric=1) all against the original table Test.
Sure enough, looking at the estimated plan (highlight the query and press Ctrl+L), we see that the query will be handled by a table scan with a predicate of
CONVERT_IMPLICIT(int,[MyDatabase].[dbo].[Test].[col],0)>={1)
AND CONVERT_IMPLICIT(int,[MyDatabase].[dbo].[Test].[col],0)<=(6)
AND isnumeric([MyDatabase].[dbo].[Test].[col])=1
Which suggests that the execution plan might be different depending on the result experienced... The question is How/why would that happen?