June 8, 2016 at 1:44 pm
Hi - very grateful for any help:
I've got the same value 2306.7555 in 2 tables,both data type float. I'm comparing the values converted to 3 decimal, but for some reason when I do that, I'm getting 2306.755 from one table and 2306.756 from the other. What's going on here?
The problem is I can't change the conversion logic because it's not mine, I'm just working with the values but I need to report this problem back to the guy responsible for the original code.
June 8, 2016 at 2:07 pm
Remember that float is an approximate numeric data type. These things can happen when using it.
You could try rounding or other options to reduce the errors.
SELECT CAST( CAST( 2306.7555 as float) AS decimal(11,3)) RawNumber,
CAST( ROUND(CAST( 2306.7555 as float), 3, 1) AS decimal(11,3)) RoundedNumber
June 8, 2016 at 3:47 pm
david.jack (6/8/2016)
Hi - very grateful for any help:I've got the same value 2306.7555 in 2 tables,both data type float. I'm comparing the values converted to 3 decimal, but for some reason when I do that, I'm getting 2306.755 from one table and 2306.756 from the other. What's going on here?
The problem is I can't change the conversion logic because it's not mine, I'm just working with the values but I need to report this problem back to the guy responsible for the original code.
There are 2 options.
1. They are not the same.
FLOAT has 15 digit precision. All 15 digits are significant.
You're displaying only 8 of them. What are the remaining 7?
2. Something with the conversion logic.
It might depend on some parameters outside of the scope.
If you post the code here we might be able to explain why the same FLOAT is converted to decimals.
_____________
Code for TallyGenerator
Viewing 3 posts - 1 through 2 (of 2 total)
You must be logged in to reply to this topic. Login to reply