February 3, 2010 at 11:33 am
I have traced a bug in my application back to using too much precision!?!?
Please compare the output of these two snippets:
declare @Foo decimal(19,8), @bar decimal(19,8), @baz decimal(19,8)
select @Foo=10000, @bar = 30000
set @baz = @Foo/@bar
select @baz
declare @f decimal(38,18), @b-2 decimal(38,18), @z decimal(38,18)
select @f=10000, @b-2 = 30000
set @z = @f/@b
select @z
The first gives 0.33333333, which is what I expected.
The second gives 0.333333000000000000 which was totally unexpected.
What I expected the second to give me was 0.333333333333333333
In my app, all ratios such as this are calculated as in the second example, which was done to give us greater precision, but in reality we're getting less precision. Can anyone please help me make sense of this and correct the issue so we get the precision we need?
Thanks in advance,
-a.
Dan Guzman - Not the MVP (7/22/2010)
All questions have to be prefaced by Server version and 'according to MS Docs' or 'my own personal opinion based on how much detail I felt like digging into at the time.'
February 3, 2010 at 2:14 pm
I just found the answer:
http://msdn.microsoft.com/en-us/library/ms190476%28SQL.90%29.aspx
I may submit this as a QOTD...
Dan Guzman - Not the MVP (7/22/2010)
All questions have to be prefaced by Server version and 'according to MS Docs' or 'my own personal opinion based on how much detail I felt like digging into at the time.'
Viewing 2 posts - 1 through 1 (of 1 total)
You must be logged in to reply to this topic. Login to reply