You cannot simply rollback a named nested transaction you can only rollback to a savepoint.

"Naming multiple transactions in a series of nested transactions with a transaction name has little effect on the transaction. Only the first (outermost) transaction name is registered with the system. A rollback to any other name (other than a valid savepoint name) generates an error. None of the statements executed before the rollback is, in fact, rolled back at the time this error occurs. The statements are rolled back only when the outer transaction is rolled back".

http://msdn.microsoft.com/en-us/library/ms188929.aspx

This example uses a savepoint to rollback a nested transaction:

CREATE TABLE MyTable (MyId INT IDENTITY (1,1),

MyCity NVARCHAR(50))

BEGIN TRANSACTION OuterTran

INSERT INTO MyTable VALUES ('Boston')

Save Transaction SavePoint1

BEGIN TRANSACTION InnerTran

INSERT INTO MyTable VALUES ('London')

ROLLBACK TRAN SavePoint1

IF (@@TRANCOUNT = 0)

BEGIN

PRINT 'All transactions were rolled back'

END

ELSE

BEGIN

PRINT 'Outer transaction is still open....rolling back...'

ROLLBACK TRANSACTION OuterTran

END

DROP TABLE MyTable

go