I'm looking for an open source ETL-Tool for creating data pipelines, but I'm cannot really find the right one. Mainly I want to use it for data transfers/aggregation between different databases (SQL-based) and different file types like csv, xlsx and xml. So the follwing things should be part of the tool:
- drag and drop conncetors/inputs (or at least a GUI for visualization)
- connectors for many different sources
- SQL and scripting (Python) for data preperation betweens steps
- open source
- good performance, should be able to handle bigger datastreams (not really big data)
- server application or at least standalone jobs
I tested the following tools:
- Pentaho ETL (no scripting/python), client is neccessary to run jobs
- Open Talend Studio (no scripting/python) (-> both open source, but commercial company backed)
- Apache nifi: nice but not really an ETL-Tool, not flexible enough for joins etc.
A typical pipeline would look like this: oracleDB-table -> SQL -> ETL-Tool -> SQL -> Python -> oracleDB-Table and to -> result.csv
Do someone know a tool to do this? Or at least a combination?
