vk-kirov (12/4/2009)
stewartc-708166 (12/3/2009)
the explanation re why the rounding off takes place is.....?Something about this can be found in BOL, topic "Precision, Scale, and Length (Transact-SQL)": http://technet.microsoft.com/en-us/library/ms190476.aspx
We have two numbers of type NUMERIC(38,10), so their precision = 38 and scale = 10.
According to the table from the above link, the result precision is: p1 - s1 + s2 + max(6, s1 + p2 + 1) = 38 - 10 + 10 + max(6, 10 + 38 + 1) = 38 + 49 = 87.
The result scale is: max(6, s1 + p2 + 1) = max(6, 10 + 38 + 1) = 49.
But there is also a note:
* The result precision and scale have an absolute maximum of 38. When a result precision is greater than 38, the corresponding scale is reduced to prevent the integral part of a result from being truncated.
Ok, the result precision (87) is definitely greater than 38, so it was reduced to 38. But why the scale was reduced to 6 – I can't find any explanation :ermm:
Thanks for this vk, it was very helpful.
As for the last part, you appaerntly overlooked applying the MAX(..) function in your calculations:
According to the table from the above link, the result precision is: p1 - s1 + s2 + max(6, s1 + p2 + 1) = 38 - 10 + 10 + max(6, 10 + 38 + 1) = 38 + 49 = 87.
should be:
1: p1 - s1 + s2 + max(6, s1 + p2 + 1)
= 38 - 10 + 10 + max(6, 10 + 38 + 1)
= 38 + max(6, 49)
= 38 + 6
= 38 *(applying the rule of absolute 38 max)
and:
The result scale is: max(6, s1 + p2 + 1) = max(6, 10 + 38 + 1) = 49.
should be:
2: max(6, s1 + p2 + 1)
= max(6, 10 + 38 + 1)
= max(6, 49)
= 6
[font="Times New Roman"]-- RBarryYoung[/font], [font="Times New Roman"] (302)375-0451[/font] blog: MovingSQL.com, Twitter: @RBarryYoung[font="Arial Black"]
Proactive Performance Solutions, Inc. [/font][font="Verdana"] "Performance is our middle name."[/font]