|
|
|
SSC Rookie
      
Group: General Forum Members
Last Login: Monday, May 13, 2013 3:20 AM
Points: 46,
Visits: 34
|
|
My answer is like this first part which is i 've answered:
Declare @value1 decimal(20,10),@value2 decimal(20,3) SET @value1 = 1234567890.123456789 SET @value2 = 0.1
SELECT @value1 * @value2
Second part is screened answer:
DECLARE @value1 DECIMAL(20,10), @value2 DECIMAL(30,13) SET @value1 = 1234567890.123456789 SET @value2 = 0.1
SELECT @value1 * @value2
Ans: The screened answer is u 've declared value2 decimal(30,13)... why u need like that? my answer is Declare @value1 decimal(20,10),@value2 decimal(20,3)...... this is enough in SQL server 2005
|
|
|
|
|
Hall of Fame
       
Group: General Forum Members
Last Login: Today @ 5:04 AM
Points: 3,046,
Visits: 1,309
|
|
pksutha (3/22/2011) My answer is like this first part which is i 've answered:
Declare @value1 decimal(20,10),@value2 decimal(20,3) SET @value1 = 1234567890.123456789 SET @value2 = 0.1
SELECT @value1 * @value2
Second part is screened answer:
DECLARE @value1 DECIMAL(20,10), @value2 DECIMAL(30,13) SET @value1 = 1234567890.123456789 SET @value2 = 0.1
SELECT @value1 * @value2
Ans: The screened answer is u 've declared value2 decimal(30,13)... why u need like that? my answer is Declare @value1 decimal(20,10),@value2 decimal(20,3)...... this is enough in SQL server 2005
That's kind of the "moral" of the question. Never use a "bigger" decimal/numeric than you actually need.
|
|
|
|
|
Ten Centuries
      
Group: General Forum Members
Last Login: Wednesday, February 06, 2013 11:51 PM
Points: 1,263,
Visits: 1,079
|
|
Duncan Pryde (3/21/2011)
michael.kaufmann (3/21/2011)
tilew-948340 (3/20/2011) [...][...] Excellent explanation. Couldn't have put it better myself.
Thank you very much for your kind words of appreciation, Duncan. Thanks again for your question and great explanation.
Regards, Michael
|
|
|
|
|
Ten Centuries
      
Group: General Forum Members
Last Login: Friday, May 17, 2013 9:52 AM
Points: 1,356,
Visits: 4,761
|
|
Duncan Pryde (3/22/2011) That's kind of the "moral" of the question. Never use a "bigger" decimal/numeric than you actually need.
The moral I've drawn is that maybe Floats aren't as bad as we thought...
|
|
|
|
|
SSC-Enthusiastic
      
Group: General Forum Members
Last Login: Thursday, February 21, 2013 11:34 AM
Points: 132,
Visits: 97
|
|
Declare @value1 numeric(38,10) Declare @value2 numeric(1,1) SET @value1 = 1234567890.123456789 SET @value2 = 0.1 SELECT @value1 SELECT @value2
SELECT @value1 * @value2 = "123456789.012345679"
|
|
|
|
|
Hall of Fame
       
Group: General Forum Members
Last Login: Today @ 5:04 AM
Points: 3,046,
Visits: 1,309
|
|
rlswisher (3/22/2011) Declare @value1 numeric(38,10) Declare @value2 numeric(1,1) SET @value1 = 1234567890.123456789 SET @value2 = 0.1 SELECT @value1 SELECT @value2
SELECT @value1 * @value2 = "123456789.012345679"
As expected.
Result precision is 38+1+1 = 40, scale is 10+1 = 11. Max allowed precision is 38, so precision and scale are reduced by 2, giving a final result precision and scale of 38,9 - which is why the result is rounded as you can see.
|
|
|
|
|
Hall of Fame
       
Group: General Forum Members
Last Login: Today @ 5:04 AM
Points: 3,046,
Visits: 1,309
|
|
Toreador (3/22/2011)
Duncan Pryde (3/22/2011) That's kind of the "moral" of the question. Never use a "bigger" decimal/numeric than you actually need. The moral I've drawn is that maybe Floats aren't as bad as we thought...
Until you want to do something like this
|
|
|
|
|
SSC-Enthusiastic
      
Group: General Forum Members
Last Login: Thursday, May 02, 2013 11:47 AM
Points: 151,
Visits: 277
|
|
Excellent question and nice explanation.. Learned something new that Precision and Scale varies for the resulting value based on (+, -, / , *, [UNION | EXCEPT | INTERSECT] , % ) .
Thanks for posting this...
|
|
|
|
|
SSCoach
         
Group: General Forum Members
Last Login: Today @ 10:25 AM
Points: 18,754,
Visits: 12,337
|
|
|
|
|
|
Ten Centuries
      
Group: General Forum Members
Last Login: Thursday, January 31, 2013 8:01 AM
Points: 1,232,
Visits: 1,046
|
|
Great question. You would not believe how long and how often programers get this wrong. There is even a list that is maintained of ROM chips that do weight conversions incorrectly becuase Dec(6,2) is used instead of Dec(13,5)
I even had to "show the math" on this exact thing a year ago when I had to explain why the weight conversion code used to change Pound to Kilos and vice versa was wrong in every appliation where I work. Not understanding this math is why so many ships have a problem balancing thier loads. Sometimes the cargo is weighed in pounds and the balast program uses Kilos. Then someone uses a cheap hand calculator to convert the 100,000 tons in pounds to Kilos and the weghts off by at least 1,000 tons.
|
|
|
|