Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase ««12

Predict output for ROUND function Expand / Collapse
Author
Message
Posted Thursday, December 31, 2009 3:01 AM


Mr or Mrs. 500

Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500

Group: General Forum Members
Last Login: Thursday, March 27, 2014 8:16 AM
Points: 534, Visits: 282
Dan Guzman - Not the MVP (12/30/2009)
Great explanation Scott, that's very detailed and instructive. Unfortunately it does not address the behavior found by KevinC. Why is not the behavior you describe consistent in a given range of numbers?

Dan


It haven't checked, but I am pretty sure it does address the behaviour found - because the binary (IEEE) representation of these numbers will not be "consistent".

Basically it boils dow to "Don't use ROUND() with double-precision"


Kelsey Thornton
MBCS CITP
Post #840741
Posted Thursday, December 31, 2009 4:39 AM
SSC Veteran

SSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC Veteran

Group: General Forum Members
Last Login: Friday, September 19, 2014 7:12 AM
Points: 291, Visits: 182
Good to know.

oracle behaves the same.

declare
round1a binary_float := 165.755;
round2a binary_float := 165.555;
round1b float := 165.755;
round2b float := 165.555;
begin
dbms_output.PUT_LINE (ROUND(round1a,2));
dbms_output.PUT_LINE (ROUND(round2a,2));
dbms_output.PUT_LINE (ROUND(round1b,2));
dbms_output.PUT_LINE (ROUND(round2b,2));
end;

165.76
165.55
165.76
165.56

Rob
Post #840774
Posted Monday, January 4, 2010 5:41 AM
SSCrazy

SSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazy

Group: General Forum Members
Last Login: Wednesday, December 17, 2014 2:36 PM
Points: 2,850, Visits: 1,165
I'll try this again. Intel processors (and probably most other computer hardware) use IEEE standard floating point representations. Single precision (REAL) uses four bytes and provides 24 bits to represent the value. Double precision (FLOAT) uses eight bytes and provides 53 bits for the value. The rounding issue discussed in this post is not due to the ROUND() function, it happens when the character string "165.755" is first converted to floating point. This means it should work the same in any programming environment on any computer hardware using the same IEEE standards.

Given the character string "165.755", the system computes the floating point value. Internally in the CPU, I believe calculations are done to 80 bit accuracy. The result is then shortened (with rounding) to 24 or 53 bits, depending on the resolution. In the case of 165.755, the 25th bit is a 1 so the single-precision value is rounded up and is a little larger than 165.755. The 54th bit is a 0, so the double-precision value is rounded down and is a little smaller than 165.755. This happens when the characters "165.755" are first converted to a floating point representation and have nothing to do with the ROUND() function.

This is why floating point types are called approximate types, and are not used where the rounding in the last decimal place is critical. If you need exact decimal values you need to use DECIMAL, NUMERIC, or MONEY. The behavior may look random, but it is completely consistent and predictable. You just need to get out paper and pencil and figure out the binary representation of all the floating point values you plan to use to 80 bits or so.



Post #841398
Posted Wednesday, January 6, 2010 3:35 AM
Ten Centuries

Ten CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen Centuries

Group: General Forum Members
Last Login: Monday, December 8, 2014 6:43 AM
Points: 1,083, Visits: 597
Nice one Scott.

What you don't know won't hurt you but what you know will make you plan to know better
Post #842647
Posted Thursday, January 7, 2010 8:51 AM
SSCrazy

SSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazy

Group: General Forum Members
Last Login: Thursday, December 11, 2014 6:27 AM
Points: 2,624, Visits: 583
Thank you for a great qotd and explanation! Lesson learned.

/Håkan Winther
MCITP:Database Developer 2008
MCTS: SQL Server 2008, Implementation and Maintenance
MCSE: Data Platform
Post #843617
Posted Friday, January 8, 2010 2:00 PM
Ten Centuries

Ten CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen Centuries

Group: General Forum Members
Last Login: Wednesday, December 17, 2014 1:55 PM
Points: 1,390, Visits: 409
Scott, very fine explantion.

-- Mark D Powell --
Post #844669
Posted Monday, January 11, 2010 10:59 AM
Ten Centuries

Ten CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen Centuries

Group: General Forum Members
Last Login: 2 days ago @ 7:41 PM
Points: 1,382, Visits: 1,753
I'm honestly surprised at the number of people here who were not aware of the approximate nature of IEEE floating-point numbers. That information was in every programming class I took and every programming book I read. (It's even in BOL: http://msdn.microsoft.com/en-us/library/ms187912.aspx -- read the "Using float and real Data" section).

As a result, the first choice I was immediately able to eliminate was the "correct" one. I have to admit, though: I didn't want to go throught the same level of work as Scott did (great explanation, by the way); nor did I want to "cheat" by copy-paste-execute, so I guessed (correctly, as it turned out).

This is not a criticism of anyone's knowledge -- it is instead a vote of applause for this question. It brings out a limitation of a commonly-used data type, of which many were not aware. This is an example of a question that both tests and educates.

Plus, it reinforces my natural instinct to avoid float whenever possible...
Post #845577
Posted Tuesday, January 12, 2010 1:30 AM
Hall of Fame

Hall of FameHall of FameHall of FameHall of FameHall of FameHall of FameHall of FameHall of FameHall of Fame

Group: General Forum Members
Last Login: Friday, May 9, 2014 12:47 AM
Points: 3,448, Visits: 4,407
sknox (1/11/2010)
I have to admit, though: I didn't want to go throught the same level of work as Scott did (great explanation, by the way); nor did I want to "cheat" by copy-paste-execute, so I guessed (correctly, as it turned out).

I cheated because I knew about the incomplete correspondence between decimals and binaries, and did not want to guess. But now I have found a great website for decimal-to-IEEE754 conversion: http://www.binaryconvert.com

This site shows hexadecimal and binary representation of an entered number, as well as the most accurate result of the conversion from this binary back to decimal.

Here is the result for 165.755 (double precision):
0x4064B828F5C28F5C
01000000 01100100 10111000 00101000
11110101 11000010 10001111 01011100
The result of backward conversion to decimal: 1.65754999999999995452526491135E2

The same for 165.555:
0x4064B1C28F5C28F6
01000000 01100100 10110001 11000010
10001111 01011100 00101000 11110110
1.65555000000000006821210263297E2

With these values the correct answer becomes obvious :)
Post #845916
Posted Tuesday, January 12, 2010 1:49 AM


Mr or Mrs. 500

Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500

Group: General Forum Members
Last Login: Thursday, March 27, 2014 8:16 AM
Points: 534, Visits: 282
Now I have found a great website for decimal-to-IEEE754 conversion: http://www.binaryconvert.com

This site shows hexadecimal and binary representation of an entered number, as well as the most accurate result of the conversion from this binary back to decimal.



Great resource - Thanks!


Kelsey Thornton
MBCS CITP
Post #845925
« Prev Topic | Next Topic »

Add to briefcase ««12

Permissions Expand / Collapse