I am currently investigating what is out there that would quickly and efficiently compare and produce a delta from CSV Files.
What I am dealing with is data that is outputted from Sybase ASE. These files are full tables and are outputted daily. They currently have up to 80 million records in any one table, with the CSV files up to 30 GB in size.
What I need to do is compare these files from one day to the next, to produce a delta which identifies new and updated records (I am currently ignoring deletes). This comparison could be between consecutive data files, or the latest data file against a dictionary or list. Currently, I am doing this in SQL server (using a custom stored procedure), but we would like to migrate away from the Microsoft stack.
Ultimately, in the end, we will be producing Parquet files from the deltas.
Initially, I have been investigating using another DBMS, such as Postgresql or even SQLite (its just a thought!), but am open to any other suggestions. I had considered using some of the big data frameworks, such as Spark, or even Python and PANDA, but am unsure whether they are the best tools for the job.
At this stage, there is a strong preference to use something open sourced, but must be able to be run on premises.
Does anyone have a suggestions on what might be a good fit?