Case statement can only have one data type that it returns. Even if the case statement has few data types that it can return, it will return a data type that all the cases can be converted to. If some of the data types are not compatibles, you'll get an error. In the example that you wrote, you should get decimal(10,3) because decimal(10,2) can be converted to decimal(10,3), but the opposite can not happen.
Adi