You cannot simply rollback a named nested transaction you can only rollback to a savepoint.
"Naming multiple transactions in a series of nested transactions with a transaction name has little effect on the transaction. Only the first (outermost) transaction name is registered with the system. A rollback to any other name (other than a valid savepoint name) generates an error. None of the statements executed before the rollback is, in fact, rolled back at the time this error occurs. The statements are rolled back only when the outer transaction is rolled back".
http://msdn.microsoft.com/en-us/library/ms188929.aspx
This example uses a savepoint to rollback a nested transaction:
CREATE TABLE MyTable (MyId INT IDENTITY (1,1),
MyCity NVARCHAR(50))
BEGIN TRANSACTION OuterTran
INSERT INTO MyTable VALUES ('Boston')
Save Transaction SavePoint1
BEGIN TRANSACTION InnerTran
INSERT INTO MyTable VALUES ('London')
ROLLBACK TRAN SavePoint1
IF (@@TRANCOUNT = 0)
BEGIN
PRINT 'All transactions were rolled back'
END
ELSE
BEGIN
PRINT 'Outer transaction is still open....rolling back...'
ROLLBACK TRANSACTION OuterTran
END
DROP TABLE MyTable
go