I'm looking for an open source ETL-Tool for creating data pipelines, but I'm cannot really find the right one. Mainly I want to use it for data transfers/aggregation between different databases (SQL-based) and different file types like csv, xlsx and xml. So the follwing things should be part of the tool:

  • drag and drop conncetors/inputs (or at least a GUI for visualization)
  • connectors for many different sources
  • SQL and scripting (Python) for data preperation betweens steps
  • open source
  • good performance, should be able to handle bigger datastreams (not really big data)
  • server application or at least standalone jobs

I tested the following tools:

  • Pentaho ETL (no scripting/python), client is neccessary to run jobs
  • Open Talend Studio (no scripting/python) (-> both open source, but commercial company backed)
  • Apache nifi: nice but not really an ETL-Tool, not flexible enough for joins etc.

A typical pipeline would look like this: oracleDB-table -> SQL -> ETL-Tool -> SQL -> Python -> oracleDB-Table and to -> result.csv

Do someone know a tool to do this? Or at least a combination?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.