Hi Ashley

Your scripts need to have an idea of the structure in the files. How many different types of format do you deal with? Is there something in each file which you can use to uniquely identify the format? If this is viable, one possible solution is:

1. Create separate directories in your Data Lake, one per file format.

2. Write a piece of code which inspects the file when it arrives, and deduces which format it contains.

3. The code can dump the file into the appropriate Data Lake directory.

4. Write a U-SQL script per directory to handle each directory's unique file format.

There are many ways of achieving this, this is just one possibility. Another option is to write something which transforms your various file formats into a common format, and then dump that up into the Data Lake.

Good luck!

Regards,

Mike.