It would be interesting to see tarsnap added to the evaluation.

cperciva · on Jan 3, 2021

Not entirely sure that I'm reproducing the benchmark correctly, but running Amazon Linux 2 I have:

    # mount -t tmpfs -o size=32G tempfs tmp
    # gdir=~/linux/.git
    # mkdir tmp/linux
    # for commit in $(git --git-dir="$gdir" rev-list v5.9 | head -n 20 | tac); do \
          mkdir ~/tmp/linux/$commit; \
          git "--git-dir=$gdir" archive "$commit" | tar -C ~/tmp/linux/$commit -xf -; \
      done
    # ~ec2-user/tarsnap-autoconf-1.0.39/tarsnap -c --dry-run --print-stats tmp
    tarsnap: Performing dry-run archival without keys
             (sizes may be slightly inaccurate)
                                           Total size  Compressed size
    All archives                          20278980504       4406609078
      (unique data)                        1767400183        258587867
    This archive                          20278980504       4406609078
    New data                               1767400183        258587867

So (assuming I reproduced the benchmark correctly!) that's 258 MB, or about 1/3 less space than bupstash.

I also had "while true; do ps aux | grep tarsnap | grep -v grep; sleep 5; done" running in a different window to monitor the RSS (I don't know what the right flags are to time(1) to get that data on Linux) and it maxed out at 11472 kB.

I suspect that Tarsnap is slower than the other options, though. That's something which I'm not able to compare easily.

If the author is interested in adding Tarsnap to his comparison, I'd love to help (and can provide a free account for benchmarking).

majewsky · on Jan 3, 2021

Since you have the well-known "ps aux | grep $THING | grep -v grep" pattern in there, I'll plug one of my favorite shell functions:

  psgrep () {
    ps aux | sed -n '1p;/\<sed\>/d;/'"$1"'/p'
  }

This does basically the same, but includes the header line from `ps aux` in the output. For instance:

  $ psgrep alacritty
  USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
  majewsky    3500  0.4  0.5 1427280 93644 ?       Sl   12:55   0:01 alacritty

cpach · on Jan 3, 2021

Interesting! Might also be useful for CSV mogrification then.

andrewchambers · on Jan 3, 2021

Really interesting to see, thanks for doing that, I feel like many fast and good rolling hash functions are yet to be invented.

array · on Jan 3, 2021

I’ve had no experience with the other tools mentioned in this post so I cant provide benchmark feedback, but just wanted to say that I’ve been a happy tarsnap customer for a couple of years now. It’s well documented and does exactly what it’s supposed to. Well done.