There are different options to solve this problem.

The easiest would be to change the text qualifier to None in your flat file Connection Manager and use a derived column or a script to strip off leading and trailing double quotes.