You can break up queries like that by doing the IsNumeric query to populate a temp table, then doing the rest of the action on the temp table, or by joining to the temp table.

-- This query separates the numbers from the text

select ID

into #T

from MyTable

where IsNumeric(MyColumn) = 1;

-- Do "math stuff" in this query

select *

from MyTable

where ID in (select ID from #T);

This avoids the "error converting" issues you're running into, and usually works well enough in terms of performance.