I've decided to go with Zstandard. I don't have exact numbers on hand, but this suits my compression needs the best, with xz
a close second. Compression times are reasonable, but I don't need to decompress my data any faster than I can process it.
Changed my mind, 7zip is even better for my needs. Here's what I did.
My Data
About 2TB of line-separated JSON objects. In other words, lots and lots of plain text.
The Command
This is the exact command I'm using to compress my data, annotated:
7z a -t7z -ms=on -myx=9 -mx=9 -mf=off -m0=PPMd:mem2g:o32 "$INPUT.7z" "$INPUT"
Now, what these mean...
7z
: The command-line tool.
a
: Adds a file to an archive (or creates if it if doesn't already exist)
-t7z
: Use an archive of type 7z (as opposed to gz
, bzip2
, lzma
, etc.)
-m
: Use these methods in compresing:
s=on
: Turn on solid mode. Might not be relevant since I'm compressing one large file (as opposed to a big group of smaller ones), but why not?
yx=9
: Do the most file analysis.
x=9
: Use the most powerful compression available.
f=off
: Turn compression filtering off. This is mainly for executable files, which I'm not processing.
0=PPMd:mem2g:o32
: Use the following parameters for the first (and in my case, only) compression method.
PPMd
: Use the PPMd algorithm, which is said to provide a "very good compression ratio for plain text files."
mem2g
: Use 2GB of RAM for compression and decompression.
o32
: Use a model order of 32. I don't honestly know what this implies, I just set it to the highest value because it felt good.
"$INPUT.7z"
: The archive I'm creating.
"$INPUT"
: The file I'm storing in the archive.
The Difference
I compressed a 219 GB subset of my data with several different programs to see which one got the best results. I wasn't benchmarking time or memory, only size. Here's what I got:
- Original File: 234,645,370,989 bytes (219 GB)
- 7zip, compressed as above: 7,201,531,161 bytes (6.8 GB)
- zstd: 7,438,787,613 bytes (7 GB)
- Command:
zstd -k -T0 -22 --ultra "$INPUT" -o "$INPUT.zst"
- lrzip: 8,531,295,280 bytes (8 GB)
- Command:
lrzip --zpaq --level=9 --maxram=40 --threads=$(nproc) -T -U "$INPUT" -o "$INPUT.lrz"
- bzip2: 20,016,871,549 bytes (19 GB)
- Command:
bzip2 --best --keep --stdout "$INPUT" > "$INPUT.bz2"
gzip: 28,807,716,394 bytes (27 GB)
- Command:
gzip --best --stdout "$INPUT" > "$INPUT.gz"
lz4: 32,455,506,529 bytes (31 GB)
- Command:
lz4 -9 -BD "$INPUT" "$INPUT.lz4"
- lzop: 34,197,587,319 bytes (32 GB)
- Command:
lzop --best --keep "$INPUT"