I am looking for a simple data integration or ETL tool to integrate data (e.g. as CSV) into our platform. There are many ETL tools (like Fivetran, Hevo, Talend, Airbyte, etc.) but they are very limited in features and mostly cover general requirements only. Enterprise solutions are too expensive for us.
What I need is (i.e. a CSV file contains a feed of product details):
- load a CSV file
- clean up and/or aggregate data
- maybe join data from other sources, like brand details, either from an API or another file
- validate for completeness, only continue for rows being seen as complete (mark them in a final report for review?)
- feed into our platform using an API
The last step is where most of the tools falls short. They mostly just offer a database destination, but the data contains e.g. images which has to be uploaded, or at least being sent as URLs to a specific API endpoint, so that the endpoint handles the upload.
I appreciate every answer helping me finding a proper solution.
