Home Forums Article Discussions Article Discussions by Author Discuss content posted by mark.stockwell Fun with Scale and Precision RE: Fun with Scale and Precision

  • December 4, 2009 at 7:34 am

    #1087720

    Explanation is in the note.

    Try to reduce the precision.

    SELECT CAST(cast(1.67574 as decimal(38,10)) / cast(10000 as decimal(38,10)) AS DECIMAL(38,10) ) conv_factor

    returns 0.0001670000

    SELECT CAST(cast(1.67574 as decimal(29,10)) / cast(10000 as decimal(29,10)) AS DECIMAL(29,10) ) conv_factor

    returns 0.0001675740