4

I have a lot of data. A lot. My data file is a little under 2TB (terabytes) of line-separated plain text records, each about 4 or 5 KB (totaling to a few hundred million records).

I'm prioritizing compression ratio, stream-ability, and decompression speed. More specifically:

  • I expect to compress this data once, then read from it many times.
  • I do not expect this data to change. I may acquire new data later, but that's a separate problem.
  • All reads from this file will be in sequential order, usually (but not always) from the beginning to the end of the file.
  • I want to back up this data somewhere. It took a long time to acquire, and I do not plan on doing it again.
  • I do not expect to uncompress this file in its entirety; only to the extent needed to pipe its contents to another program.
  • I don't care about preserving file metadata.

This on a Linux HPC cluster, so I'm looking for a command-line utility. My main contenders are gzip, bzip2, and lzma, but I'm open to other options. It's okay if I have to compile software myself, as long as I don't need root access.

1

I would seriously consider taking a look at the HDF5 format as it is specifically designed for this sort of use-case. Details of supported compression formats can be found here but include pre-defined ZLIB and SZIP plus several 3rd party options.

The reference software includes a number of language bindings and there are bindings for python available via pip.

  • It looks like HDF5 is primarily for numerical data? My data isn't really numerical, it's textual. – JesseTG Mar 8 '18 at 10:57
  • @JesseTG: HDF5 can combine almost any data types, e.g. hdfgroup.org/portfolio-item/medicine where images, metadata, text & numbers are all being stored. – Steve Barnes Mar 8 '18 at 18:58
  • And which utility is it, exactly? I'm only finding documentation for the C/C++ APIs (which are not what I want right now). – JesseTG Mar 8 '18 at 20:48
0

I've decided to go with Zstandard. I don't have exact numbers on hand, but this suits my compression needs the best, with xz a close second. Compression times are reasonable, but I don't need to decompress my data any faster than I can process it.

Changed my mind, 7zip is even better for my needs. Here's what I did.

My Data

About 2TB of line-separated JSON objects. In other words, lots and lots of plain text.

The Command

This is the exact command I'm using to compress my data, annotated:

7z a -t7z -ms=on -myx=9 -mx=9 -mf=off -m0=PPMd:mem2g:o32 "$INPUT.7z" "$INPUT"

Now, what these mean...

  • 7z: The command-line tool.
  • a: Adds a file to an archive (or creates if it if doesn't already exist)
  • -t7z: Use an archive of type 7z (as opposed to gz, bzip2, lzma, etc.)
  • -m: Use these methods in compresing:
    • s=on: Turn on solid mode. Might not be relevant since I'm compressing one large file (as opposed to a big group of smaller ones), but why not?
    • yx=9: Do the most file analysis.
    • x=9: Use the most powerful compression available.
    • f=off: Turn compression filtering off. This is mainly for executable files, which I'm not processing.
    • 0=PPMd:mem2g:o32: Use the following parameters for the first (and in my case, only) compression method.
      • PPMd: Use the PPMd algorithm, which is said to provide a "very good compression ratio for plain text files."
      • mem2g: Use 2GB of RAM for compression and decompression.
      • o32: Use a model order of 32. I don't honestly know what this implies, I just set it to the highest value because it felt good.
  • "$INPUT.7z": The archive I'm creating.
  • "$INPUT": The file I'm storing in the archive.

The Difference

I compressed a 219 GB subset of my data with several different programs to see which one got the best results. I wasn't benchmarking time or memory, only size. Here's what I got:

  • Original File: 234,645,370,989 bytes (219 GB)
  • 7zip, compressed as above: 7,201,531,161 bytes (6.8 GB)
  • zstd: 7,438,787,613 bytes (7 GB)
    • Command: zstd -k -T0 -22 --ultra "$INPUT" -o "$INPUT.zst"
  • lrzip: 8,531,295,280 bytes (8 GB)
    • Command: lrzip --zpaq --level=9 --maxram=40 --threads=$(nproc) -T -U "$INPUT" -o "$INPUT.lrz"
  • bzip2: 20,016,871,549 bytes (19 GB)
    • Command: bzip2 --best --keep --stdout "$INPUT" > "$INPUT.bz2"
  • gzip: 28,807,716,394 bytes (27 GB)

    • Command: gzip --best --stdout "$INPUT" > "$INPUT.gz"
  • lz4: 32,455,506,529 bytes (31 GB)

    • Command: lz4 -9 -BD "$INPUT" "$INPUT.lz4"
  • lzop: 34,197,587,319 bytes (32 GB)
    • Command: lzop --best --keep "$INPUT"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.