0

I am currently investigating what is out there that would quickly and efficiently compare and produce a delta from CSV Files.

What I am dealing with is data that is outputted from Sybase ASE. These files are full tables and are outputted daily. They currently have up to 80 million records in any one table, with the CSV files up to 30 GB in size.

What I need to do is compare these files from one day to the next, to produce a delta which identifies new and updated records (I am currently ignoring deletes). This comparison could be between consecutive data files, or the latest data file against a dictionary or list. Currently, I am doing this in SQL server (using a custom stored procedure), but we would like to migrate away from the Microsoft stack.

Ultimately, in the end, we will be producing Parquet files from the deltas.

Initially, I have been investigating using another DBMS, such as Postgresql or even SQLite (its just a thought!), but am open to any other suggestions. I had considered using some of the big data frameworks, such as Spark, or even Python and PANDA, but am unsure whether they are the best tools for the job.

At this stage, there is a strong preference to use something open sourced, but must be able to be run on premises.

Does anyone have a suggestions on what might be a good fit?

Thanks

  • If you already have a satisfactory process working in SQL Server, migrating it to Postgres seems like the simplest process. Are there some additional requirements or limitations that make that an untenable solution? – rd_nielsen Jun 18 at 14:47
  • Postgres is probably where I am leaning, but I think we want to take the opportunity to explore other tools and opportunity. While we currently can't call our data repository "big data" yet, we want to explore the possibility of these tools. – Ash Jun 18 at 22:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.