So you need to cast it like this?
round(cast(5 as float) / cast(3 as float),1)
As ziangij already says, casting one of the operands is sufficient. And if you do use explicit casting, I'd suggest you to cast to the datatype of the result: decimal(5,2) instead of float.
You can also use implicit casting: ROUND(5.0/3, 1)
The additional .0 forces SQL Server to treat is as numeric. (Even the trailing zero can be omitted, but for readability I prefer "5.0" over "5.")
And to follow up on a promise I made earlier to people who prefer code in a copy/paste-able format, here it is:
DECLARE @Result decimal(5,2);
SET @Result = ROUND(5/3, 1);
Hugo Kornelis, SQL Server/Data Platform MVP (2006-2016)
Visit my SQL Server blog: http://sqlblog.com/blogs/hugo_kornelis