• Tom.Thomson (5/31/2010)


    But I'm a bit surprised you found these ones complex.

    With hindsight, I could have and should have worked it out by hand. But as you said - the horizontal scrolling and parenthesis counting made my head hurt. If I had checked the site later in the day, I probably would have done it by hand.

    If you really want accuracy, don't you have to use CAST(60 as numeric(10,5)) to keep the error below 0.00006? Using CAST(60 as numeric(10,4)) does the division and truncates the result; subsequently that is rounded but this has no effect as it's rounding a numeric(10,4) value to numeric(10,4). (Another reason to dislike SQL's approach to type conversion, truncation, and rounding for expressions on the RHS of assignments.)

    When dividing by numeric(10,4), the result is not numeric(10,4). The rules for precision and scale of the result of arithmetic operations are quite complex. And documented in Books Online.


    Hugo Kornelis, SQL Server/Data Platform MVP (2006-2016)
    Visit my SQL Server blog: https://sqlserverfast.com/blog/
    SQL Server Execution Plan Reference: https://sqlserverfast.com/epr/