I have an application that has different inputs. Most of them are txt files or CSV/Excel Files. They contain the same type of data, though they are formatted in different ways (order of columns different, sometimes they are separated with spaces, or tabs).

How can I achieve a sort of automation that, based on the knowledge that I know the type of output data, parses the input data "normalizing" them into the same format?

What technologies do I need to look for?


Import each of the tables into a database, clean each one as appropriate, join them with a UNION query, and export them to CSV or whatever format your application will accept. This process can all be automated with a SQL script. The import and export processes will need DBMS-specific functions, but those can be automated also.

Of course, you need to be able to write SQL (or hire somebody to do so) to take this approach.

  • Hello, this is a manual step, that requires that I do already know the format of the input. What I was looking for was knowing if there is any kind of solution (for sure data scientist should be able to know how, I believe) that I might start studying to reach this goal. ML can be helpful, but I am not sure if that's the correct solution. Thank you!
    – Pliskin
    Sep 17 '19 at 12:37

Not the answer you're looking for? Browse other questions tagged or ask your own question.