I had this problem once before, but I was using a slightly different formula for calculating distances between lat/long pairs. Unfortunately, I no longer work for my employer at the time and don't have the notes I made of this issue to be more specific, so I only vaguely recollect the circumstances. IIRC, I discovered that there were certain input values that would cause the T-SQL COS function to return results just slightly outside the -1 < x < 1 range. This cascaded through the calculation and resulted in an input to the ACOS function that also fell outside the -1 < x < 1 range, which generated the domain error. I corrected the problem by adjusting any results of the COS function that fell outside the range to 1 or -1 as appropriate.

The way I finally diagnosed the problem was by running a query that included a column for each individual part of the formula, building up from column to column until I had the whole result. When I ran this query on the dataset that caused the domain error, I narrowed down the query to include only the rows that caused the error. Then, I reran the query, taking columns out of the query (starting from the whole formula and working down to the pieces) one by one until I didn't get the error any more. At that point, I looked at the results and noticed the out-of-range result for the COS function.

I've tried a few things to see if I could replicate the out-of-range result for the COS function on my SQL Server 2008R2 instance, but haven't been able to do so.