SP2 is very old. Is applying SP4+CU3+MS11-049 an option?

One more question about the data. How many rows are you calculating the Median over? Median being what it is, you have to store every value before you can calculate the answer meaning your object may be asking for a lot of memory and I have also seen reports of a high volume of memory allocations coming from SQLCLR becoming an issue at times. The Sort in Terminate is also going to be very expensive. I wrote a Median aggregate in SQLCLR not too long ago as a proof-of-concept (POC) and IIRC I used a sorted collection so the class sorted values into the collection as they were read. I am not sure how this compares to your technique in terms of performance, but I can share my code if you're interested. Mine never made it beyond the POC and I never really hammered it from a performance perspective. SQL Server may not be serializing your object, which is why rewriting that method had no affect one way or the other. Just some thoughts.