0

I am currently investigating what is out there that would quickly and efficiently compare and produce a delta from CSV Files.

What I am dealing with is data that is outputted from Sybase ASE. These files are full tables and are outputted daily. They currently have up to 80 million records in any one table, with the CSV files up to 30 GB in size.

What I need to do is compare these files from one day to the next, to produce a delta which identifies new and updated records (I am currently ignoring deletes). This comparison could be between consecutive data files, or the latest data file against a dictionary or list. Currently, I am doing this in SQL server (using a custom stored procedure), but we would like to migrate away from the Microsoft stack.

Ultimately, in the end, we will be producing Parquet files from the deltas.

Initially, I have been investigating using another DBMS, such as Postgresql or even SQLite (its just a thought!), but am open to any other suggestions. I had considered using some of the big data frameworks, such as Spark, or even Python and PANDA, but am unsure whether they are the best tools for the job.

At this stage, there is a strong preference to use something open sourced, but must be able to be run on premises.

Does anyone have a suggestions on what might be a good fit?

Thanks

2
  • If you already have a satisfactory process working in SQL Server, migrating it to Postgres seems like the simplest process. Are there some additional requirements or limitations that make that an untenable solution?
    – rd_nielsen
    Jun 18 '19 at 14:47
  • Postgres is probably where I am leaning, but I think we want to take the opportunity to explore other tools and opportunity. While we currently can't call our data repository "big data" yet, we want to explore the possibility of these tools.
    – Ash
    Jun 18 '19 at 22:24
0

The program languages that can be used to process large files are as follows:

  1. Conventional high-level programming languages, such as Java, C/C++, C#, Basic, etc.
  2. The file data is imported into the database and processed by SQL
  3. Python
  4. esProc SPL

Taking text file as an example, this paper introduces the characteristics of the above methods for large file calculation .

2
  • What's your recommendation?
    – Alejandro
    Dec 15 '20 at 4:04
  • That's four answers. Please post them separately so that they may be voted upon in their own right.
    – Chenmunka
    Dec 16 '20 at 14:37
0

you can try Acho Studio it works pretty well with larger CSV files

1
  • Can you detail why this program complies with the question request?
    – Alejandro
    Jan 12 at 2:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.