Your brute force method can't possibly be giving you the correct int result.

Given that each byte of the varbinary is 8 bits of an integer you would have to multiple each one by some factor of 2 before adding them together. Even then you would need to know if the varbinary representation is big-endian or little endian.