I got some hints when I tried another file, it seems the first couple characters in the raw data file has some non-numeric character so the script throws error like:
Msg 245, Level 16, State 1, Line 1
Conversion failed when converting the varchar value '326567 ' to data type int.
Those three characters are EF BB EF, what are they for? the file is in ANSI/ASCII format.
I also tried to save the raw file in a different format UTF-8, same error.
If I save the file as Unicode, EF BB EF were removed, but then I got error:
Msg 8114, Level 16, State 5, Line 1
Error converting data type varchar to bigint.
As I can inspect each character in hex code, I suspect that the hex 20 (which means blank space) is causing the issue