The "rounding" differences are actually taking place when the numbers are stored as real and float approximate numbers. These are IEEE standard formats, they are in fact documented. You can look at the binary values with this query:

SELECTF24, ROUND(F24, 2), CAST(F24 AS VARBINARY(8)),

F25, ROUND(F25, 2), CAST(F25 AS VARBINARY(8))

FROM (SELECT CAST(165.755 AS FLOAT(24)) AS F24, CAST(165.755 AS FLOAT(25)) AS F25) x

FLOAT(1) through FLOAT(24) is stored as 4-byte single-precision floating point, or REAL. FLOAT(25) through FLOAT(53) is stored as 8-byte double-precision floating point.

The 8-byte version is 0x4064B828F5C28F5C. 406 is the sign bit and exponent, 4B828F5C28F5C is the mantissa. (There are 53 mantissa bits but the leading bit is always 1 and not stored, so the next 52 bits are shown as 13 hex digits). The exact value would be 4B828F5C28F5C28F5C (with 28F5C repeating forever), but it has to stop and round at the 13th character. The 54th bit of the mantissa is 0, so the remainder is discarded and what is stored is something like 165.754999999999. This rounding occurs as the characters "165.755" are converted to FLOAT, the ROUND() function comes later.

The 4-byte version is 0x4325C148. The sign bit and exponent take 9 bits (43 plus part of the 2), so the mantissa looks different from above because it is shifted one bit. If you divide 4B828F5C28F5C (followed by 28F5C forever) by 2 (or convert to binary, tack a extra 0 on the front, and regroup the bits into hex digits) you get 25C147AD147AD (followed by 147AD forever). When this is rounded off after 24 bits, the remainder starts with a 1 bit so the final 7 is rounded up to 8. You end up with roughly 165.75501.

So the ROUND() function is not rounding the exact value 165.755 in either case. With single-precision it is rounding something slightly larger and with double precision it is rounding something slightly smaller.