Predict output for ROUND function

SQLServerCentral is supported by Red Gate Software Ltd.

Popular Topics

Home

Members

Calendar

Who's On

Home » Article Discussions » Article Discussions by Author » Discuss content posted by Suresh Babu Kari » Predict output for ROUND function

19 posts, Page 2 of 2

««1 2

Predict output for ROUND function

Rate Topic

Display Mode

Topic Options

Author

Message

Kelsey Thornton

Posted Thursday, December 31, 2009 3:01 AM

Mr or Mrs. 500

Group: General Forum Members
Last Login: Monday, March 04, 2013 4:10 AM
Points: 532, Visits: 281

Dan Guzman - Not the MVP (12/30/2009)

Great explanation Scott, that's very detailed and instructive. Unfortunately it does not address the behavior found by KevinC. Why is not the behavior you describe consistent in a given range of numbers?

Dan

It haven't checked, but I am pretty sure it does address the behaviour found - because the binary (IEEE) representation of these numbers will not be "consistent".

Basically it boils dow to "Don't use ROUND() with double-precision"

Kelsey Thornton
MBCS CITP

Post #840741

Rob de Vos

Posted Thursday, December 31, 2009 4:39 AM

SSC Veteran

Group: General Forum Members
Last Login: Tuesday, May 21, 2013 2:18 AM
Points: 287, Visits: 141

Good to know.

oracle behaves the same.

declare
round1a binary_float := 165.755;
round2a binary_float := 165.555;
round1b float := 165.755;
round2b float := 165.555;
begin
dbms_output.PUT_LINE (ROUND(round1a,2));
dbms_output.PUT_LINE (ROUND(round2a,2));
dbms_output.PUT_LINE (ROUND(round1b,2));
dbms_output.PUT_LINE (ROUND(round2b,2));
end;

165.76
165.55
165.76
165.56

Rob

Post #840774

Scott Coleman

Posted Monday, January 04, 2010 5:41 AM

SSCrazy

Group: General Forum Members
Last Login: Tuesday, May 21, 2013 1:39 PM
Points: 2,818, Visits: 1,038

I'll try this again. Intel processors (and probably most other computer hardware) use IEEE standard floating point representations. Single precision (REAL) uses four bytes and provides 24 bits to represent the value. Double precision (FLOAT) uses eight bytes and provides 53 bits for the value. The rounding issue discussed in this post is not due to the ROUND() function, it happens when the character string "165.755" is first converted to floating point. This means it should work the same in any programming environment on any computer hardware using the same IEEE standards.

Given the character string "165.755", the system computes the floating point value. Internally in the CPU, I believe calculations are done to 80 bit accuracy. The result is then shortened (with rounding) to 24 or 53 bits, depending on the resolution. In the case of 165.755, the 25th bit is a 1 so the single-precision value is rounded up and is a little larger than 165.755. The 54th bit is a 0, so the double-precision value is rounded down and is a little smaller than 165.755. This happens when the characters "165.755" are first converted to a floating point representation and have nothing to do with the ROUND() function.

This is why floating point types are called approximate types, and are not used where the rounding in the last decimal place is critical. If you need exact decimal values you need to use DECIMAL, NUMERIC, or MONEY. The behavior may look random, but it is completely consistent and predictable. You just need to get out paper and pencil and figure out the binary representation of all the floating point values you plan to use to 80 bits or so.

Post #841398

BudaCli

Posted Wednesday, January 06, 2010 3:35 AM

Ten Centuries

Group: General Forum Members
Last Login: Wednesday, March 27, 2013 9:02 AM
Points: 1,046, Visits: 573

Nice one Scott.

What you don't know won't hurt you but what you know will make you plan to know better

Post #842647

hakan.winther

Posted Thursday, January 07, 2010 8:51 AM

SSCrazy

Group: General Forum Members
Last Login: Monday, May 06, 2013 5:31 AM
Points: 2,226, Visits: 438

Thank you for a great qotd and explanation! Lesson learned.

/Håkan Winther
MCITP:Database Developer 2008

Post #843617

Mark D Powell

Posted Friday, January 08, 2010 2:00 PM

Ten Centuries

Group: General Forum Members
Last Login: Tuesday, May 21, 2013 12:26 PM
Points: 1,258, Visits: 341

Scott, very fine explantion.

-- Mark D Powell --

Post #844669

sknox

Posted Monday, January 11, 2010 10:59 AM

Ten Centuries

Group: General Forum Members
Last Login: Tuesday, May 21, 2013 9:31 AM
Points: 1,041, Visits: 1,356

I'm honestly surprised at the number of people here who were not aware of the approximate nature of IEEE floating-point numbers. That information was in every programming class I took and every programming book I read. (It's even in BOL: http://msdn.microsoft.com/en-us/library/ms187912.aspx -- read the "Using float and real Data" section).

As a result, the first choice I was immediately able to eliminate was the "correct" one. I have to admit, though: I didn't want to go throught the same level of work as Scott did (great explanation, by the way); nor did I want to "cheat" by copy-paste-execute, so I guessed (correctly, as it turned out).

This is not a criticism of anyone's knowledge -- it is instead a vote of applause for this question. It brings out a limitation of a commonly-used data type, of which many were not aware. This is an example of a question that both tests and educates.

Plus, it reinforces my natural instinct to avoid float whenever possible... Tongue

Post #845577

vk-kirov

Posted Tuesday, January 12, 2010 1:30 AM

Hall of Fame

Group: General Forum Members
Last Login: Yesterday @ 6:30 AM
Points: 3,192, Visits: 4,151

sknox (1/11/2010)

I have to admit, though: I didn't want to go throught the same level of work as Scott did (great explanation, by the way); nor did I want to "cheat" by copy-paste-execute, so I guessed (correctly, as it turned out).

I cheated because I knew about the incomplete correspondence between decimals and binaries, and did not want to guess. But now I have found a great website for decimal-to-IEEE754 conversion: http://www.binaryconvert.com

This site shows hexadecimal and binary representation of an entered number, as well as the most accurate result of the conversion from this binary back to decimal.

Here is the result for 165.755 (double precision):
0x4064B828F5C28F5C
01000000 01100100 10111000 00101000
11110101 11000010 10001111 01011100
The result of backward conversion to decimal: 1.65754999999999995452526491135E2

The same for 165.555:
0x4064B1C28F5C28F6
01000000 01100100 10110001 11000010
10001111 01011100 00101000 11110110
1.65555000000000006821210263297E2

With these values the correct answer becomes obvious :)

Post #845916

Kelsey Thornton

Posted Tuesday, January 12, 2010 1:49 AM

Mr or Mrs. 500

Group: General Forum Members
Last Login: Monday, March 04, 2013 4:10 AM
Points: 532, Visits: 281

Now I have found a great website for decimal-to-IEEE754 conversion: http://www.binaryconvert.com

This site shows hexadecimal and binary representation of an entered number, as well as the most accurate result of the conversion from this binary back to decimal.

Great resource - Thanks! Cool

Kelsey Thornton
MBCS CITP

Post #845925

« Prev Topic | Next Topic »

19 posts, Page 2 of 2

««1 2

Permissions