If you are certain that the first number always denotes the start of the reference number, then you can substring it using patindex as start marker and take 4 chars from there...
declare @string varchar(100)
set @string = 'First and Second 1234 Avenue'
select SUBSTRING(@string, PATINDEX('%[0-9]%', @string), 4)
----
1234
(1 row(s) affected)
/Kenneth