After reading a lot of manual pages, I can recommend the following:
As it has in-built integrity check adjustable to SHA-256 using the following switch:
You may specify the number of CPU threads using the following switch:
The default compression level switch is set to 6, you may change that to the maximum with:
It implements a maximum effort algorithm using the following switch, though it is rather slow:
You can easily define an alias of whatever combination you choose.
I will add to this section sequentially what I have tested out.
xz (multi-threading + SHA-256 + level 9 + extreme)
alias xz-extreme='xz --format=xz --check=sha256 -9 --threads=8 --keep --verbose --verbose --extreme'
With this command the compression took almost 25 minutes and resulted in 0.772 ratio.
$ xz-extreme data.tar
7,991.0 MiB / 10.1 GiB = 0.772 7.0 MiB/s 24:28
xz (multi-threading + CRC-32 + default level 6)
alias xz='xz --format=xz --threads=8 --keep --verbose --verbose'
With this command the compression took less than 15 minutes and resulted in 0.786 ratio (+138 MiB).
$ xz data.tar
8,129.8 MiB / 10.1 GiB = 0.786 12 MiB/s 14:15
In contrast to
xz, due to its single-threaded nature, the same file with the command:
$ bzip2 --keep -9 --verbose data.tar
data.tar: 1.244:1, 6.430 bits/byte, 19.63% saved, 10848870400 in, 8719149908 out.
Took exactly 27 minutes while achieving 0.803 ratio (+324MiB).
Short data analysis of the presented data set
Further un-compressible files 4.3 GiB (like RAR or 7-Zip on max compression).
Image (JPEG) files 2 GiB.
Windows executable installers (means also compressed) 0.8 GiB.
Only the remaining 3 GiB were more or less well compressible files.
Different data set
about a 100,000 code files (highly compressible data)
some PDFs containing JPEG images of text (rather un-compressible)
Interesting results summary
- 4.5G code_and_some_books.tar
- 2.3G code_and_some_books.tar.gz (
- 2.1G code_and_some_books.tar.bz2 (
- 1.6G code_and_some_books.tar.xz (the above
xz-extreme alias used)