The problem was with REPLICATE truncating the data before it ever got to CLR.

Per the documentation:

If string_expression is not of type varchar(max) or nvarchar(max), REPLICATE truncates the return value at 8,000 bytes. To return values greater than 8,000 bytes, string_expression must be explicitly cast to the appropriate large-value data type.

DECLARE @val nvarchar(MAX)

DECLARE @val2 nvarchar(MAX)

SET @val = N'A'

SET @val2 = REPLICATE(@val,8002)

SELECT dbo.BigHash('md5',@val2)

while

DECLARE @val2 nvarchar(MAX)

SET @val2 = REPLICATE(N'A',8002)

SELECT dbo.BigHash('md5',@val2)

does not.