sqldriver (2/17/2014)
mickyT (2/16/2014)
HiI've never compared my processes against a power shell equivalent (or anything else for that matter:-)), but I have some large text files (around 1.5 GB) that I pre-process before I load them. I use PERL for this. It handles regular expressions and large files without any real issues. Essentially my process does something similar to yours. Read in a line (or more), run some replaces then write to a new file.
I would give you some timings for my conversions, but I don't time them individually and they are also working on zipped files adding a bit more overhead.
I can't say PERL would be quicker, but it could be worth looking at.
I prefer to look at PERL through a telescope. With my eyes closed and my fingers jammed in my ears 😀
Fair enough 🙂