On: Zstandard – Real-time data compression algorithm

[petre]

I did some tests myself on a 22MB SQL file and it turns out:

* 7za -m0=PPMd produced the smallest file being faster than bzip2

* bzip2 turned out to be way faster than both lz (684%) and xz (644%) and produced a smaller file

* xz is marginally faster than lz, compressed sizes are about the same with the xz file being a tad smaller

* without any switches 7za produces an archive a bit bigger than xz and lzip in about the same amount of time

* gzip and zst produce about the same compressed size, only zstd is a lot faster (517%) than gzip

The 7z file was produced using the -m0=PPMd switch. For the other files no command line switches were supplied. Here are the file sizes:

  23668150 file.sql
   3899477 file.sql.7z
   4149962 file.sql.bz2
   5954982 file.sql.gz
   4540628 file.sql.lz
   4506720 file.sql.xz
   5961291 file.sql.zst

[dorfsmay → petre]

When going for smallest size, it'd be interesting to see your comparison using command lines switches for best compression (makes a big difference, both in terms of time and size).

Was bzip2 slightly, or considerably slower than zst?

[petre → dorfsmay]

Bzip2 being slower than gzip, yes, it's also considerably slower than zstd. Yet zstd -19 produced a bigger (4.3M) file in about the same amount of time.

If I can remember correctly zstd = 0.2s, gzip = 0.8s, 7zip (PPMd) = 2.1s, bzip2 = 2.7s, lzip, xz, 7zip (lzma) = 15..16s. This is CPU time from memory, might not be fully accurate.

I'd say zstd and gzip is better suited for general use, while bzip2 and 7zip (PPMd) are better suited for high compression of text files.