March 11, 2009 at 9:00 am
Hello
Thanks For Last Answer & Your Guide .
But I want Multiple This Number ???
declare @N1 decimal(19,7)
declare @N2 decimal(19,7)
declare @N3 decimal(19,7)
declare @N Decimal(31,7)
declare @f decimal(38,8)
----------123456789012 1234567
Set @N1 = 999999999999.6666667
Set @N2 = 999999999999.3333334
Set @N3 = 999999999999.5555556
Set @N = @N1 * @N2
Select @N
Set @f = @N / @N3
Select @f
--Answer Is =======> 999999999999.44444440
--With Calculator ===> 999999999999.44444449999997530865
How Can Get This Answer ???
Thanks
Habibi
March 11, 2009 at 10:04 am
You are only allowing 7 digits of precision right of the decimal point. Your SQL results are returning 7 digits of precision right of the decimal point, which is one more than you have a right to expect, given that you did a division in there.
If you want more precision in your answers then you will need to allow more precision in your operands.
[font="Times New Roman"]-- RBarryYoung[/font], [font="Times New Roman"] (302)375-0451[/font] blog: MovingSQL.com, Twitter: @RBarryYoung[font="Arial Black"]
Proactive Performance Solutions, Inc. [/font][font="Verdana"] "Performance is our middle name."[/font]
March 11, 2009 at 10:16 am
Actually, I just checked and it now appears to me that your SQL answer is correct and your calculator is incorrect at the 8th place.
[font="Times New Roman"]-- RBarryYoung[/font], [font="Times New Roman"] (302)375-0451[/font] blog: MovingSQL.com, Twitter: @RBarryYoung[font="Arial Black"]
Proactive Performance Solutions, Inc. [/font][font="Verdana"] "Performance is our middle name."[/font]
March 11, 2009 at 4:06 pm
01AV01 - May I ask why you want/need the calculator answer? Especially when
In science and engineering, convention dictates that unless a margin of error is explicitly stated, the number of significant figures used in the presentation of data should be limited by the precision of those data. For example, if one instrument can read to tenths of a unit of measurement, calculations related to data obtained from that instrument can only be confidently stated to the tenths place, regardless of what the raw calculation returns or even if other data used in the calculation can be obtained more precisely. Even outside these disciplines, there is a tendency to assume that all the non-zero digits of a number are meaningful; thus, providing excessive figures may lead the viewer to expect better precision than actually exists.
Source of above quotation
http://en.wikipedia.org/wiki/False_precision
If you reason is to just learn why / how a computer processes decimal numbers use wikipedia and search for "multiplication precision" and read the some hundreds of pages explaining why.
Viewing 4 posts - 1 through 4 (of 4 total)
You must be logged in to reply to this topic. Login to reply