Mostly Helpful Stuff

Free Carrots #4: Making Backups

Free Carrots is an ongoing series of helpful tips for using Plan 9.

How does one perform incremental backups of the files stored on a Plan 9 file system? Haha, why would anyone ever need to do that?

Many years ago, as I contemplated migrating from UNIX to Plan 9, I needed a replacement for rsync. Fortunately, my transition was taking place while those of us trying to get 9front off the ground were doing some work on the installer and ISO, and I found myself face to face with an existing utility that covered about 90% of what I needed from rsync.

mkfs(8) copies files from one tree to another, copying only the files that are out of date (literally, when the timestamps don’t match), and of those, only the files that are included in the specified proto(2) file.

For example, this is how copydist in the 9front installer copies files from the install media to the user’s disk:

   ; disk/mkfs -z 16372 -U -s /n/dist -d /n/newfs /sys/lib/sysconfig/proto/allproto

This particular proto file contains only +, which copies every single file in the source tree to the destination tree. (This will be useful later.)

Perhaps more instructive is the proto file used to build the 9front ISO, /sys/lib/sysconfig/proto/distproto. It follows the same simple format, starting at the root of the tree and working its way down, specifying owner, group, and permissions at an arbitrarily granular level, like so:

   mode=ug+rw
   mode=o-w
   uid=sys
   gid=sys
   adm    d775 adm adm
          uid=adm
          gid=adm
          timezone      d775
                 *
   cfg    d775
          pxe    d775
   cron   d775
   lib    d775
          *
          audio  d775
                 icon   d775
                        *
          cmap   d775
                 *
          dict   d775
                 *
          face   d775
                 +

And so on.

It turned out that for my purposes, simply copying everything was enough. This is my one-line rsync replacement,

   ; disk/mkfs $args -v -z 4096 -s $1 -d $d <{echo +} >[2=1] | tee -a $log

extracted from my fully-operational, and very site-specific script:

http://plan9.stanleylieber.com/rc/backup

I’ve been using some variation of this script for going on ten years. Launched from cron(8) or executed manually for “quick” copying. It’s not exactly fast (see also: copydist), but it has served reliably for backing up website, mailing list, and other important data, preserving continuity over the years through several catastrophic server failures, and other, unnamed calamaties.

I agree. Slow is not great. Much faster is this third-party program written by kvik:

https://git.sr.ht/~kvik/clone

which uses parallel processes, similar to Plan 9’s existing fcp(1) command, but unlike fcp it also knows how to copy directories. Unfortunately, it does not yet know how to compare timestamps.

Help?

Update:

qwx has written a shell script that uses derp(1), awk(1), and tar(1) to copy changed files. The bonus here is that it also removes files that have been deleted from the source directory:

http://shithub.us/qwx/rc/HEAD/bin/syncab/raw

cat vs carrot toy