ISNUMERIC doesn't guarantees that the value is of NUMERIC datatype.

It's only the indicator if the value can be converted into one of many SQL Server numeric data types (exact or approximate ones).

Your sample is easily converted into money:

SELECT CAST(',1234' AS MONEY)

Actually, MONEY is one of the least "restrictive" numeric datatype, as a lot of complete rubbish can still be converted into it:

SELECT CAST('$' AS MONEY)

SELECT CAST('.' AS MONEY)

SELECT CAST(',' AS MONEY)

SELECT CAST('.,' AS MONEY)

SELECT CAST(',,,,.' AS MONEY)

SELECT CAST('-,.' AS MONEY)

All the above will work, therefore, ISNUMERIC will return true for all of the above...

SQL2012 offers the new TRY_CONVERT function, but pre-2012 there is nothing "out of the box" for this sort of thing.

However, you can create your own function (SQL or CLR) to do so, depending on exact data types you want to validate.