The first thing is you don't seem to quite understand CASE expressions. There is often little need to NEST them like you have done here. If you have a different expression to resolve you can just add an extra WHEN (no need for CASE). We can, however, simplify that section a little more, as you're testing for 0 and then >0 but evaluate the same result. Thus this will do the same:
CASE WHEN (ISNULL(Roylaty,0) + ISNULL(advance,0)) >= 0 THEN 0
     ELSE ISNULL(Roylaty,0) + ISNULL(Advance,0) + ISNULL(Rights,0)
END as unearnedadv

I wouldn't have, however, said that you SQL was overly complex. What are you looking to simplify exactly? it is quite concise in all honesty (after changing the CASE expression). If you wanted, you could change the Sub query into a WITH, but they are very similar.