I sympathize with your situation. I have no technical suggestions because as you have suggested with the data changing there are no reliable solutions to the problem. Here are some thoughts to consider.
- Revisit the exports out of the source system and try to get an actual file useful in data processing
- Make this an operational not a development task. Maybe someone has to input this into a data entry screen. Alternatively maybe someone needs to normalize the file before passing it on to data processing
- As developers we should strive to build automation with very low failure rates. Explain to your manager that anything built to do this will have very high failure rates
- If you as the developer are forced to provide operational support for tasks like this then at one point you will not have time to develop since you will be constantly fixing things.
Good luck!
Hi Chrissy,
Thanks for the tips! I figured I'd post the issue here just in case I might be missing something. I just don't think there is a way to automate this and will explore some of your suggestions.