I have an Operational Data Store (ODS) in MSSQL Server 2014 which consists of Data fetched from multiple source systems (ODBC - NetSuite, JDBC - MySQL, Postgres, API - Salesforce, Jira) using a combination of Java and ETL tool Talend.

After data is loaded in ODS, I want to validate the result with above Source Systems. I tried to use the Java JDBC code to read and write Data from JDBC source systems to CSV. But not all JDBC drivers support the function/property of setFetchSize, and without it reading and writing data to CSV for millions of records is time-consuming. Plus even if there is a workaround for using setFetchSize still it won't be useful for non-Jdbc type SourceSystems like the API ones mentioned above.

So I started looking for Tools which would do Data Validation between Source and Target. Couldn't find any Open Source Tool, but found a couple of Licensed Tools (QuerySurge, ETL Validator) but again this tools are either too expensive or does not support all types of Source Systems.

Since I am familiar with Java and Talend so I wanted to know if there is any workaround for this issue. If not then can anyone please suggest a Tool for Data Validation which will support all types of Source System and is Open Source/Licensed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.