4

I am looking for a lossless archive/compression library that supports random-access extraction of particular files, and would also like to perform such extraction in C++, meaning from C++ executable:

  1. Load the entire archive to memory
  2. Perform extraction for particular files from the archive in memory to a buffer in memory.

The purpose is to reduce the number of times I have to read data from the drive, and also the amount of data transferred from the drive.

I know that such algorithms exist - for example, windows "zip" and some tar compression schemes are able to do this. However, what I need is to perform these operations from a C++ executable, and step 1 above is important so as to reduce the number of times I have to read/transfer data from the drive - so performing command line operations within C++ is not a sufficient solution.

Example:

  1. I have 3 archived files, and and I know I have to read 7 files located among 2 of the archives.
  2. I read archives 1 & 2 into memory (the ones I know that contain the 7 files).
  3. I selectively extract 3 files from archive 1 (I know that they are in this archive).
  4. I selectively extract 4 files from archive 2 (I know that they are in this archive).
  5. Done.

Explanation: In this example I managed to reduce the amount of times I have to read from the drive from 7 to 2 times. If I used os/cmd operations, I would have read directly from the drive and this would be like reading 7 times. So such a solution is not sufficient. My use case requires me to reduce the number of times I go to the drive to read data.

Any suggestions? Thanks.

Improve this question
10
  • why don't you just use the filesystem's built-in transparent compression, like the one in NTFS, ZFS or Btrfs?
    – phuclv
    Commented Mar 1, 2020 at 10:06
  • 1
    @phuclv, can you explain more clearly? I am unfamiliar with what you are recommending. Also, I think this solution seems to be overlooking my comment above: "reduce the number of times I have to read ... so performing command line operations within C++ is not a sufficient solution." I will add another clarification in case this was unclear.
    – AOK
    Commented Mar 1, 2020 at 12:00
  • 1
    @phuclv, I added an example, which I hope, makes what I am looking for clearer.
    – AOK
    Commented Mar 1, 2020 at 12:07
  • 1
    transparent compression means a compressed file looks just like a normal file and any bytes can be accessed randomly, the filesystem driver will handle which block to decompress. If you load the archives to memory then extract some files inside it to memory then it's not random access in the archive because the files have already been extracted fully
    – phuclv
    Commented Mar 1, 2020 at 14:18
  • This is terrible difficult to answer, as it depends on a lot of things: file size, access pattern,.. Loading it in memory will deplete RAM. This can have adverse effects. My suggestion would be to find another approach. Maybe look into memory mapped files [1] and see if this helps solving your problem. [1] en.wikipedia.org/wiki/Memory-mapped_file
    – Jens
    Commented Mar 4, 2020 at 14:20

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.