Post reply

select decimals after calculations, weird display

  • October 2, 2013 at 7:05 pm

    #299314

    Hi, I can't get why I have that strange results, I need to keep value of dividing in a column let say 1/35, it's displays as 0 no matter what kind of convert/cast I using,

    however if I add 35.0 like for a1 then it works.

    Internally all this results are columns in intermediate temp tables derived as a result of sum() or count(), no any no any special formatting done. c1, c2 alwayse int, but I need very presise result in "a" column. Later I will use "a" in ssrs .

    select SUM(1) c1, SUM(35) c2 into #t

    select c1, c2,

    CAST(1/35.0 as decimal(8,6)) a1,

    CAST(1/35 as decimal(8,6)) a2,

    CAST(c1/c2 as decimal(8,6)) a3

    from #t

    c1 c2 a1 a2 a3

    1 35 0.0285710.0000000.000000

    Tx

    Mario

  • October 2, 2013 at 9:09 pm

    #1655375

    CAST before you divide:

    SELECT CAST(c1 AS DECIMAL(8,6)) / CAST(c2 AS DECIMAL(8,6))

    FROM #t;

    Or not:

    SELECT 1.*c1 / c2

    FROM #t;

    My mantra: No loops! No CURSORs! No RBAR! Hoo-uh![/I]

    My thought question: Have you ever been told that your query runs too fast?

    My advice:
    INDEXing a poor-performing query is like putting sugar on cat food. Yeah, it probably tastes better but are you sure you want to eat it?
    The path of least resistance can be a slippery slope. Take care that fixing your fixes of fixes doesn't snowball and end up costing you more than fixing the root cause would have in the first place.

    Need to UNPIVOT? Why not CROSS APPLY VALUES instead?[/url]
    Since random numbers are too important to be left to chance, let's generate some![/url]
    Learn to understand recursive CTEs by example.[/url]
    [url url=http://www.sqlservercentral.com/articles/St

  • October 3, 2013 at 1:10 am

    #1655415

    To give a little more information about dwain's answer (he is totally correct):

    you are doing an integer division before you cast it to a decimal.

    This means the result is rounded to zero before you cast.

    Need an answer? No, you need a question
    My blog at https://sqlkover.com.
    MCSE Business Intelligence - Microsoft Data Platform MVP

  • October 3, 2013 at 7:56 am

    #1655566

    dwain.c (10/2/2013)

    CAST before you divide:

    SELECT CAST(c1 AS DECIMAL(8,6)) / CAST(c2 AS DECIMAL(8,6))

    FROM #t;

    Or not:

    SELECT 1.*c1 / c2

    FROM #t;

    You can also cheat if you're not too worried about precision (more on that in a minute)...

    SELECT (c1+0.0)/ c2 FROM #t;

    If you are worried about precision (and I normally am) during Division or Multiplication of Decimal numbers, like when you're calculating periodic interest and the "whole" must come out to the correct penny, then Dwain's method will work the best except that you must increase the both the scale and the precision of the conversions by CAST. Do your calculations with a large precision (I recommend at least 15 decimal places like what a calculator does) and then do a final cast for display purposes if the display requires less precision.

    --Jeff Moden

    RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
    First step towards the paradigm shift of writing Set Based code:
    ________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

    Change is inevitable... Change for the better is not.

    Helpful Links:
    How to post code problems
    How to Post Performance Problems
    Create a Tally Function (fnTally)

  • October 3, 2013 at 9:40 am

    #1655616

    Thanks, all

    Thansk, Dwaine

    M

Viewing 5 posts - 1 through 4 (of 4 total)

You must be logged in to reply to this topic. Login to reply