Archive format that allows for global ("solid") compression, but with the ability to efficiently extract individual files

Ask Question

Asked 5 months ago

Modified 5 months ago

Viewed 11 times

I want to:

Archive millions of tiny files ...
- Each file is about 1-3KB of size
- Plain UTF-8 text
- Contents largely similar (logfiles)
... into larger archives ...
- Say, 1 days worth of files in each archive (about 2GB uncompressed)
... with "solid" compression ...
- To save on storage costs (we're hosting in The Cloud) I want to be able to efficiently compress the files, which means that the compression should look for patterns across files, not just compress each file individually. This is sometimes known as a "solid" archive, although the name might vary.
... and be able to efficiently extract single files.
- I wish to build a tool which allows a human to locate and inspect individual files inside the archives, so I want to be able to extract individual files relatively quickly. All archive formats that I'm familiar with that have "solid" mode available (7zip, RAR, TAR+gzip/bzip2/whatever) require that all previous files be decompressed as well, since the compression just treats all individual files as one single large data stream. This means that files at the start are quick to access, while the files at the end of the archive will require decompressing (and discarding) the entire archive before the last file can be reconstructed.
Compression speed doesn't matter. It can be slow. It can require lots of disk space or RAM. What matters is that the end result is compact and quick to access randomly.

Ideally this would be available as a library for PHP or NodeJS, however command line tools are acceptable too.

asked May 3 at 13:39

Vilx-

1115 bronze badges

Common compression algorithms build dynamic dictionaries of data parts they compress. If you want to compress data over multiple files the dictionary has to be built over multiple files, but for decompressing the same dictionary state is necessary and as the dictionary content changes dynamically while compression data the same has to be performed when decompressing. This is why solid archives always need to be processed as a whole. IMHO as it is just 2GB of text you should try non-solid archive formats like ZIP, RAR and check how far you get.
– Robert
May 7 at 16:44
@Robert That's why I feel I might need something uncommon. :) Alternatively, I just realized that as a compromise I can just "group" the files together manually (like in TAR files) and then compress those groups with a common file format. Or vice versa - compress groups of files as individual solid archives, and then TAR those archives together in a single file for storage.
– Vilx-
May 7 at 21:36
I just noticed that in the recent stable 7-zip UI app, when creating a 7-Zip archive there is an option "solid block size". by default this option is set to "4 GB" but you can change it to down to "1 MB". I have not tried it but this sounds to me like the "groups of files as individual solid archives" you were talking about.
– Robert
May 10 at 10:10
@Robert - Hey, yeah, that sounds good!
– Vilx-
May 10 at 11:26

Add a comment |

0 Your Answer

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.