https://www.simple-talk.com/sql/sql-tools/sqlxml-bulk-loader-basics/

I came across the article above today and the author says he deals with importing tens of gigs at a time. I think XML might be ideal for this situation since it can handle "missing" or variable number of columns. Of course you'll need to preprocess your flat file into an XML file. The author claims it is the quickest and most versatile way of bulk loading huge amounts of data.