I have the following task to do
- Read several hundred XML and JSON files from various sources (HTTP, FTP, ...) with a size from a 100kb to 900mb.
- The files slowly changes and have to be read several times a day.
- The files contain documents (nodes with multiple key / value attributes) which need to be uniformed (most files have the same structure but they may differ).
- For each change in a document (Create, Update and Remove) a call to a HTTP rest API needs to be made.
- A overview of which files are read successfully and which failed should be show.
I'm looking looking for a software (or library, preferred for PHP or Python) which runs on a server (AWS) to handle this in a easy way.
My searches for such software ended up in ETL's like Matillion which could handle the collecting part (but not really confortable) and struggles with the exporting part (as it is build to write all data in a database).
Before i build some thing specifically, is there a general solution available?