Which compression utility should I use for an extremely large plain text file?

Question

I have a lot of data. A lot. My data file is a little under 2TB (terabytes) of line-separated plain text records, each about 4 or 5 KB (totaling to a few hundred million records).

I'm prioritizing compression ratio, stream-ability, and decompression speed. More specifically:

• I expect to compress this data once, then read from it many times.
• I do not expect this data to change. I may acquire new data later, but that's a separate problem.
• All reads from this file will be in sequential order, usually (but not always) from the beginning to the end of the file.
• I want to back up this data somewhere. It took a long time to acquire, and I do not plan on doing it again.
• I do not expect to uncompress this file in its entirety; only to the extent needed to pipe its contents to another program.
• I don't care about preserving file metadata.

This on a Linux HPC cluster, so I'm looking for a command-line utility. My main contenders are gzip, bzip2, and lzma, but I'm open to other options. It's okay if I have to compile software myself, as long as I don't need root access.

Answer 1

I've decided to go with Zstandard. I don't have exact numbers on hand, but this suits my compression needs the best, with xz a close second. Compression times are reasonable, but I don't need to decompress my data any faster than I can process it.

Changed my mind, 7zip is even better for my needs. Here's what I did.

My Data

About 2TB of line-separated JSON objects. In other words, lots and lots of plain text.

The Command

This is the exact command I'm using to compress my data, annotated:

7z a -t7z -ms=on -myx=9 -mx=9 -mf=off -m0=PPMd:mem2g:o32 "$INPUT.7z" "$INPUT"

Now, what these mean...

• 7z: The command-line tool.
• a: Adds a file to an archive (or creates if it if doesn't already exist)
• -t7z: Use an archive of type 7z (as opposed to gz, bzip2, lzma, etc.)
• -m: Use these methods in compresing:
  • s=on: Turn on solid mode. Might not be relevant since I'm compressing one large file (as opposed to a big group of smaller ones), but why not?
  • yx=9: Do the most file analysis.
  • x=9: Use the most powerful compression available.
  • f=off: Turn compression filtering off. This is mainly for executable files, which I'm not processing.
  • 0=PPMd:mem2g:o32: Use the following parameters for the first (and in my case, only) compression method.
    • PPMd: Use the PPMd algorithm, which is said to provide a "very good compression ratio for plain text files."
    • mem2g: Use 2GB of RAM for compression and decompression.
    • o32: Use a model order of 32. I don't honestly know what this implies, I just set it to the highest value because it felt good.
• "$INPUT.7z": The archive I'm creating.
• "$INPUT": The file I'm storing in the archive.

The Difference

I compressed a 219 GB subset of my data with several different programs to see which one got the best results. I wasn't benchmarking time or memory, only size. Here's what I got:

• Original File: 234,645,370,989 bytes (219 GB)
• 7zip, compressed as above: 7,201,531,161 bytes (6.8 GB)
• zstd: 7,438,787,613 bytes (7 GB)
  • Command: zstd -k -T0 -22 --ultra "$INPUT" -o "$INPUT.zst"
• lrzip: 8,531,295,280 bytes (8 GB)
  • Command: lrzip --zpaq --level=9 --maxram=40 --threads=$(nproc) -T -U "$INPUT" -o "$INPUT.lrz"
• bzip2: 20,016,871,549 bytes (19 GB)
  • Command: bzip2 --best --keep --stdout "$INPUT" > "$INPUT.bz2"

• gzip: 28,807,716,394 bytes (27 GB)

  • Command: gzip --best --stdout "$INPUT" > "$INPUT.gz"

• lz4: 32,455,506,529 bytes (31 GB)

  • Command: lz4 -9 -BD "$INPUT" "$INPUT.lz4"
• lzop: 34,197,587,319 bytes (32 GB)
  • Command: lzop --best --keep "$INPUT"

Answer 2

I would seriously consider taking a look at the HDF5 format as it is specifically designed for this sort of use-case. Details of supported compression formats can be found here but include pre-defined ZLIB and SZIP plus several 3rd party options.

The reference software includes a number of language bindings and there are bindings for python available via pip.