I'm looking for an open source ETL-Tool for creating data pipelines, but I'm cannot really find the right one. Mainly I want to use it for data transfers/aggregation between different databases (SQL-based) and different file types like csv, xlsx and xml. So the follwing things should be part of the tool:

  • drag and drop conncetors/inputs (or at least a GUI for visualization)
  • connectors for many different sources
  • SQL and scripting (Python) for data preperation betweens steps
  • open source
  • good performance, should be able to handle bigger datastreams (not really big data)
  • server application or at least standalone jobs

I tested the following tools:

  • Pentaho ETL (no scripting/python), client is neccessary to run jobs
  • Open Talend Studio (no scripting/python) (-> both open source, but commercial company backed)
  • Apache nifi: nice but not really an ETL-Tool, not flexible enough for joins etc.

A typical pipeline would look like this: oracleDB-table -> SQL -> ETL-Tool -> SQL -> Python -> oracleDB-Table and to -> result.csv

Do someone know a tool to do this? Or at least a combination?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.