FreeBSD tips#1: Turn back time, with automated filesystem snapshots

I’m not a sysadmin, but I do have to administer a number of systems.  This inevitably involves learning many bits of obscure arcana, which are all too easily forgotten once the heat of battle has passed.  So, I’ve decided I’d do well to make notes for myself, and if any of them end up being useful for other system hackers out there, so much the better.

Tip #1 – Turn back time, with automated filesystem snapshots

Wouldn’t it be nice to be able to undo our mistakes?  Well, with my first script, you can.  FreeBSD’s filesystem, UFS2, supports a very nice feature called filesystem snapshots.  With a single command, you can make a complete, instant-in-time, snapshot of your filesystem, in just a few seconds. What’s more, it uses virtually no no disk space. (Actually that’s not quite true, but let’s not get bogged down in that). The point is, it’s gives us a great mechanism to save us from ourselves.

The raw snapshot feature, mksnap_ffs (8), doesn’t quite fit our bill though.  There are a couple of problems.

1. A snapshot only covers one mount point. If we want to snapshot our entire filesystem, we need to snapshot each mount point, e.g. /, /usr, /var, /home and so on.
2. A snapshot is just an opaque file. To browse it, it must first be attached to a device and then mounted.

So, I decided to cooked up a shell script which creates snapshots for all the mount points on the chosen device (e.g. ad0), creates a mirror set of mount points under /snapped_fs/, attaches the snapshots to memory disk devices, and mounts them on the mirror mount points.  So I always have “yesterday’s” filesystem to hand in case I need to recover a deleted file, undo changes, or run a diff.

WARNING!!! This script runs as root AND messes with your entire filesystem!  It could easily do massive and untold DAMAGE. It works for me, but who knows, it may nuke your ENTIRE system! It is YOUR responsibility to satisfy yourself that is ok to use.  If you come crying to me I WILL just laugh heartily!

Remember, this is just a hack script. It is not production quality.  There is no man page.  There is no -h option.  It won’t validate your inputs, it won’t inform you that the inputs you entered are not in the exact format it understands (but will carry on anyway), it won’t ask you if you’re really sure, and it won’t give you shiny white teeth.  But it may just save your bum one day.

So what are you waiting for? Get snapping!!  Download snap_fs.sh

Usage

snap_fs.sh takes exactly two arguments. The first is the directory where you want the mirror to be created. I usually go for “/snapped_fs“.  The second is an expression which should match the physical device(s) you want to snapshot. This is basically used to extract mount points from df (1) using sed (1).

So in my case, I run:

$./snap_fs.sh /snapped_fs "/dev/ad[0-3]"

IMPORTANT!!! Parameter 1 must not have a trailing slash (although it might work anyway, I’ve no idea). If the directory already exists, it might get wiped or the script might baulk or the sky may fall down.  I haven’t tested that either.  Parameter 2 need to be wrapped in quotes if you’re using anything other than a literal pattern.  Check the script if you’re unsure.

If it works, you should find that the output of df (1) looks something like this:

/dev/ad0s1a     507630    237418   229602    51%    /
devfs                1         1        0   100%    /dev
/dev/ad0s1d    1012974    117978   813960    13%    /tmp
/dev/ad0s1f    8122126   4747258  2725098    64%    /usr
/dev/ad0s1e    4058062   1790056  1943362    48%    /var
/dev/ad0s1g   60986306     36234 56071168     0%    /home
fdescfs              1         1        0   100%    /dev/fd
/dev/ad1     1240358296 155746416 65383220    70%    /home/sim
/dev/md0        507630    236586   230434    51%    /snapped_fs
/dev/md1      60986306        10 56107392     0%    /snapped_fs/home
/dev/md2     240358296 154817428 66312208    70%    /snapped_fs/home/sim
/dev/md3       1012974    117066   814872    13%    /snapped_fs/tmp
/dev/md4       8122126   4510986  2961370    60%    /snapped_fs/usr
/dev/md5       4058062   1218216  2515202    33%    /snapped_fs/var

snap_fs.sh knows how to unmount, detach and delete the previous snapshot before it creates the new one, so you just run it whenever you like.  Add it to the system crontab with something like this:

0  1 * * * /bin/sh /usr/local/etc/scripts/cron/snap_fs.sh /snapped_fs "/dev/ad[0-1]"

Taking it further

This is all good stuff, but the real power of filesystem snapshot is to use them as the basis of a system backup.  Using rsnapshot on another server, you can take a nightly remote images of your system, secure in the knowledge that no matter how long the transfer takes, the files represent a single, coherent, instant in time.  That will be the subject of another article.