A Fix() Function in T-SQL

Question

Post reply

A Fix() Function in T-SQL

Rob Farley

SSCrazy

Points: 2738
More actions
June 27, 2006 at 1:45 pm

#168987

Comments posted to this topic are about the content posted at http://www.sqlservercentral.com/columnists/rfarley/afixfunctionintsql.asp
Rob Farley
LobsterPot Solutions & Adelaide SQL Server User Group
Company: http://www.lobsterpot.com.au
Blog: http://blogs.lobsterpot.com.au

Viewing 9 posts - 1 through 8 (of 8 total)

You must be logged in to reply to this topic. Login to reply

David.Poole SSC Guru Points: 75897 More actions · Answer 1

Solution for constipated mathematicians, work it out with paper and pencil, if that doesn't work use LOGs

LinkedIn Profile

Hugo Kornelis SSC Guru Points: 64780 More actions · Answer 2

Nice function.

Tip 1: Instead of the CASE to get rid of negative values, use ABS.

Tip 2: Add a CASE to work around the domain error you'd get from LOG10(0).

Tip 3: Stop being lazy and use decimal instead of float. Because float can't represent many values exactly, you run the risk of errors. For instance:

SELECT dbo.fix(1.15) -- Returns 1.1 instead of 1.2

alter

function dbo.fix(@num numeric(36,18), @digits int) returns numeric(36,18) as

begin

declare @res float

select @res = case when @num = 0 then 0

else round(@num,@digits-1-floor(log10(abs(@num)))) end

return (@res)

end

Hugo Kornelis, SQL Server/Data Platform MVP (2006-2016)
Visit my SQL Server blog: https://sqlserverfast.com/blog/
SQL Server Execution Plan Reference: https://sqlserverfast.com/epr/

jwainz Right there with Babe Points: 739 More actions · Answer 3

This article is quite a coincidence, I was just thinking this morning, how could I use log? Now if I could just remember how they worked, it's been 14 years since I've last used them.

Ian Yates SSCoach Points: 19738 More actions · Answer 4

Having completed IT + Maths degrees at university it is nice to see someone using some maths in the real world - I reckon 99/100 SQL developers would've used a varchar conversion with something to cater for the possible decimal point and minus signs and lived with the performance loss. Nice article and good thinking!

ScottPletcher SSC Guru Points: 100947 More actions · Answer 5

Very sweet!

But why the extra overhead of declaring and setting a variable? Why not just return the calc'd value?

...

BEGIN

RETURN CASE WHEN ...

END

SQL DBA,SQL Server MVP(07, 08, 09) A socialist is someone who will give you the shirt off *someone else's* back.

Skip Hansen SSC Rookie Points: 49 More actions · Answer 6

Nice to see someone using LOGs. But I would trust your results more if you displayed them, instead of using the WHERE expected_result <> actual_result.

The problem I found was that you don't get exact representation when using float. Frankly, I have never used float before, so maybe I am doing something incorrectly, but note the example below.

declare @num float

, @res float

, @digits int

SET @num = 1.23456678

SET @digits = 4

select @res = case when @num > 0

then round(@num,@digits-1-floor(log10(@num)))

else round(@num,@digits-1-floor(log10(-@num)))

end

select @num as ORIGINAL_NUMBER, @res AS RESULT

, @digits as NBR_SIG_DIGITS

RESULT IS:

ORIGINAL_NUMBER RESULT NBR_SIG_DIGITS

1.23456678 1.2349999999999999 4

Probably not what you really want (I assume you want 1.235 ??)

Hugo Kornelis SSC Guru Points: 64780 More actions · Answer 7

Hi Skip,

>>The problem I found was that you don't get exact representation when using float<<

Of course you don't - check Books Online. Float is an approximate-number data type. For most numbers, it can only store a (close) approximization.

Internally, float uses base-2 representation. As a result, only numbers that can be written as x + (y / EXP(2, z)) where x and y are integers and z is a non-negative integer can be represented exactly in that representation. Just like our common base-10 representation can only represent an approximation of 1/3, float can only represent an approximation of 1.235.

That's one of the reasons why I wrote that the function should use decimal instead of float in my previous reply (though I now see that I forgot to change the datatype for the internal variable).

Hugo Kornelis, SQL Server/Data Platform MVP (2006-2016)
Visit my SQL Server blog: https://sqlserverfast.com/blog/
SQL Server Execution Plan Reference: https://sqlserverfast.com/epr/

Rob Farley SSCrazy Points: 2738 More actions · Answer 8

True, I should've used numeric. I said in the article it was out of laziness.

I chose float originally because that's the type that the trigonometry functions use. Ideally I'd have it return the same type as is passed in (the way that the others work). But I knew numeric would be better... I was just being lazy.

As for using abs... yes - that would've been better. And I can't believe I didn't see that it doesn't work for 0. That's awful of me.

Thanks Hugo...

Rob

Rob Farley
LobsterPot Solutions & Adelaide SQL Server User Group
Company: http://www.lobsterpot.com.au
Blog: http://blogs.lobsterpot.com.au