I agree working with dates from text files is not particularly great in SSIS but I have done the following in the past.

Set the text file connection manager to import the column as a string, then add a derived column transformation to your data flow. Finally add an expression to replace the value of the date column something like the following which converts emtpy strings to nulls.

(DT_STR, 50, 1252)(LEN([Datefield]) == 0 ? NULL(DT_WSTR, 50) : [Datefield])