I think your case is about distributed transaction (linked server). If so, about your case, I think a restart is needed to complete the rollback.

I guess, the difference is, rollback in a restart is a regular local one. Without restart, rollback is a process by Microsoft Distributed Transaction Coordinator (MS DTC). But I'm not so sure.

BTW, it might not help, but I suggest you SET XACT_ABORT ON.