1

As the title says I am in search for a good, defined as the best available, general-purpose command-line compression program with good integrity check in Linux.

These archives will be used for long term storage

on Blu-ray M-Discs; details on used hardware.

Requirements:

  • the better in-built integrity check, the better software for my archiving purposes

  • must be open-source

  • preferably available in packaged form in most distributions, or better = installed by default

  • adjustable compression ratio, while using up to 10GB RAM when compressing

  • should be multi-threaded as I have 8 core CPU

Improve this question
4
  • Should the integrity check only be a check (for instance including a MD5 in the filename), or should it attempt correction?
    – Nicolas Raoul ♦
    Commented Jun 27, 2018 at 12:23
  • @NicolasRaoul If I understand your question correctly, it would be nice to have the option to attempt recovery.
    – Vlastimil Burián
    Commented Jun 27, 2018 at 12:29
  • checking file integrity and changing compression ratio are the most basic things that all archiving tools must have
    – phuclv
    Commented Jun 27, 2018 at 15:02
  • @phuclv I agree, but this question is to what degree certain tools have it.
    – Vlastimil Burián
    Commented Jun 27, 2018 at 15:03

1 Answer 1

Reset to default
2

After reading a lot of manual pages, I can recommend the following:

xz (man page)

As it has in-built integrity check adjustable to SHA-256 using the following switch:

--check=sha256

Further bonuses:

  • You may specify the number of CPU threads using the following switch:

    --threads=8

  • The default compression level switch is set to 6, you may change that to the maximum with:

    -9

  • It implements a maximum effort algorithm using the following switch, though it is rather slow:

    --extreme

You can easily define an alias of whatever combination you choose.

Benchmarks

I will add to this section sequentially what I have tested out.

  1. xz (multi-threading + SHA-256 + level 9 + extreme)

    alias xz-extreme='xz --format=xz --check=sha256 -9 --threads=8 --keep --verbose --verbose --extreme'

    With this command the compression took almost 25 minutes and resulted in 0.772 ratio.

    $ xz-extreme data.tar

7,991.0 MiB / 10.1 GiB = 0.772   7.0 MiB/s      24:28

  2. xz (multi-threading + CRC-32 + default level 6)

    alias xz='xz --format=xz --threads=8 --keep --verbose --verbose'

    With this command the compression took less than 15 minutes and resulted in 0.786 ratio (+138 MiB).

    $ xz data.tar

8,129.8 MiB / 10.1 GiB = 0.786    12 MiB/s      14:15

bzip2 (man page)

In contrast to xz, due to its single-threaded nature, the same file with the command:

$ bzip2 --keep -9 --verbose data.tar

data.tar:  1.244:1,  6.430 bits/byte, 19.63% saved, 10848870400 in, 8719149908 out.

Took exactly 27 minutes while achieving 0.803 ratio (+324MiB).

Short data analysis of the presented data set

  • Further un-compressible files 4.3 GiB (like RAR or 7-Zip on max compression).

  • Image (JPEG) files 2 GiB.

  • Windows executable installers (means also compressed) 0.8 GiB.

  • Only the remaining 3 GiB were more or less well compressible files.

Different data set

  • about a 100,000 code files (highly compressible data)

  • some PDFs containing JPEG images of text (rather un-compressible)

Interesting results summary

  1. 4.5G code_and_some_books.tar
  2. 2.3G code_and_some_books.tar.gz (-9)
  3. 2.1G code_and_some_books.tar.bz2 (-9)
  4. 1.6G code_and_some_books.tar.xz (the above xz-extreme alias used)
Improve this answer
2
  • I've tried it on 6.1MB plain text files on a dual-core machine. tar -cf zzz.tar * && bzip2 -9 zzz.tar took 1.5s and resulted in a 968k file, while the same with xz using your options took 11s and resulted in an 863k file. So xz took 7 times as long while offering about 11% better compression with that input. My conclusion: if in a frequent process I want to (temporarily) save some space, I'd rather go with bzip2 – while for long-time archival, xz might be the better choice. Though this conclusion is based on too little data :)
    – Izzy
    Commented Jun 27, 2018 at 14:23
  • Thanks a lot for the effort you took (upvoted now). I leave my corrected comment above for an additional example. In everyday life most of us rarely compress such huge amounts as you just did :)
    – Izzy
    Commented Jun 27, 2018 at 16:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.