I suspect that the cause of the problem is that you have a row in the data where TextDescription begins with 'Int.nr.TXT:' but there is no valid integer after.
This is a perfect time to use one of the new functions in SQL 2012: TRY_CAST()
TRY_CAST will return null if there was a problem with the cast instead of generating an error.
Your code could then be rewritten as:
UPDATEtblInvoicesProject
SETTTT_Factureren = 0,
TTT_FactuurStatus = 0,
TTT_FactuurNummer = 0
WHERETTT_ProjectID = 26
AND TTT_FactuurStatus = 1
AND TTT_ID NOT IN(
SELECT isnull(TRY_CAST(SUBSTRING(CAST(X.TextDescription AS VARCHAR(100)),13,10) as int), -1)
FROM [500].dbo.frstx X
WHERE (LEFT(CAST(X.TextDescription AS VARCHAR(100)), 11) = 'Int.nr.TXT:')
)
Note the use of isnull to convert any failed casts from null to -1. I assume that -1 is not a valid project number.
If you have a null value in NOT IN () the results will probably not be what you expect.