FreeBSD tips#1: Turn back time, with automated filesystem snapshots

I’m not a sysadmin, but I do have to administer a number of systems.  This inevitably involves learning many bits of obscure arcana, which are all too easily forgotten once the heat of battle has passed.  So, I’ve decided I’d do well to make notes for myself, and if any of them end up being useful for other system hackers out there, so much the better.

Tip #1 – Turn back time, with automated filesystem snapshots

Wouldn’t it be nice to be able to undo our mistakes?  Well, with my first script, you can.  FreeBSD’s filesystem, UFS2, supports a very nice feature called filesystem snapshots.  With a single command, you can make a complete, instant-in-time, snapshot of your filesystem, in just a few seconds. What’s more, it uses virtually no no disk space. (Actually that’s not quite true, but let’s not get bogged down in that). The point is, it’s gives us a great mechanism to save us from ourselves.

The raw snapshot feature, mksnap_ffs (8), doesn’t quite fit our bill though.  There are a couple of problems.

1. A snapshot only covers one mount point. If we want to snapshot our entire filesystem, we need to snapshot each mount point, e.g. /, /usr, /var, /home and so on.
2. A snapshot is just an opaque file. To browse it, it must first be attached to a device and then mounted.

So, I decided to cooked up a shell script which creates snapshots for all the mount points on the chosen device (e.g. ad0), creates a mirror set of mount points under /snapped_fs/, attaches the snapshots to memory disk devices, and mounts them on the mirror mount points.  So I always have “yesterday’s” filesystem to hand in case I need to recover a deleted file, undo changes, or run a diff.

WARNING!!! This script runs as root AND messes with your entire filesystem!  It could easily do massive and untold DAMAGE. It works for me, but who knows, it may nuke your ENTIRE system! It is YOUR responsibility to satisfy yourself that is ok to use.  If you come crying to me I WILL just laugh heartily!

Remember, this is just a hack script. It is not production quality.  There is no man page.  There is no -h option.  It won’t validate your inputs, it won’t inform you that the inputs you entered are not in the exact format it understands (but will carry on anyway), it won’t ask you if you’re really sure, and it won’t give you shiny white teeth.  But it may just save your bum one day.

So what are you waiting for? Get snapping!!  Download snap_fs.sh

Usage

snap_fs.sh takes exactly two arguments. The first is the directory where you want the mirror to be created. I usually go for “/snapped_fs“.  The second is an expression which should match the physical device(s) you want to snapshot. This is basically used to extract mount points from df (1) using sed (1).

So in my case, I run:

$./snap_fs.sh /snapped_fs "/dev/ad[0-3]"

IMPORTANT!!! Parameter 1 must not have a trailing slash (although it might work anyway, I’ve no idea). If the directory already exists, it might get wiped or the script might baulk or the sky may fall down.  I haven’t tested that either.  Parameter 2 need to be wrapped in quotes if you’re using anything other than a literal pattern.  Check the script if you’re unsure.

If it works, you should find that the output of df (1) looks something like this:

/dev/ad0s1a     507630    237418   229602    51%    /
devfs                1         1        0   100%    /dev
/dev/ad0s1d    1012974    117978   813960    13%    /tmp
/dev/ad0s1f    8122126   4747258  2725098    64%    /usr
/dev/ad0s1e    4058062   1790056  1943362    48%    /var
/dev/ad0s1g   60986306     36234 56071168     0%    /home
fdescfs              1         1        0   100%    /dev/fd
/dev/ad1     1240358296 155746416 65383220    70%    /home/sim
/dev/md0        507630    236586   230434    51%    /snapped_fs
/dev/md1      60986306        10 56107392     0%    /snapped_fs/home
/dev/md2     240358296 154817428 66312208    70%    /snapped_fs/home/sim
/dev/md3       1012974    117066   814872    13%    /snapped_fs/tmp
/dev/md4       8122126   4510986  2961370    60%    /snapped_fs/usr
/dev/md5       4058062   1218216  2515202    33%    /snapped_fs/var

snap_fs.sh knows how to unmount, detach and delete the previous snapshot before it creates the new one, so you just run it whenever you like.  Add it to the system crontab with something like this:

0  1 * * * /bin/sh /usr/local/etc/scripts/cron/snap_fs.sh /snapped_fs "/dev/ad[0-1]"

Taking it further

This is all good stuff, but the real power of filesystem snapshot is to use them as the basis of a system backup.  Using rsnapshot on another server, you can take a nightly remote images of your system, secure in the knowledge that no matter how long the transfer takes, the files represent a single, coherent, instant in time.  That will be the subject of another article.

Misfeatures #1

From http://dev.mysql.com/doc/refman/5.0/en/extensions-to-ansi.html:

MySQL Server understands the || and && operators to mean logical OR and AND, as in the C programming language. In MySQL Server, || and OR are synonyms, as are && and AND. Because of this nice syntax, MySQL Server doesn’t support the standard SQL || operator for string concatenation; use CONCAT() instead. Because CONCAT() takes any number of arguments, it’s easy to convert use of the || operator to MySQL Server.

Which brain-dead moron came up with this gem of a misfeature?   What’s C got to do with anything? (And I like C).  What they meant to say was “Because of this nice syntax, seasoned users of other systems will waste their time trawling through the docs to find out why on earth a most basic SQL standard doesn’t work as it should.” I shall limit myself to saying that this example epitomises the MySQL philosophy, and is frankly the kind of mindset more commonly associated with Redmond.

NO2ID – *Really* saying NO to the database state

My previous post reminded me of another screengrab I caught back in 2006.  I’m all for the NO2ID campaign, but perhaps they were taking it a bit far on this occasion:

Date: 2006-02-27, 22:24

Net Neutrality

I had to laugh when I took a look at Wikipedia’s entry for Editor War last month. I saved the screenie for future amusement value (and in case they ever do resolve the dispute. Yeah right…)

Nerdy but amusingly matter-of-fact statements #1

The first in a throw-away series of matter-of-fact statements about incomprehensibly large or improbable concepts:

 #1 – From Wikipedia’s entry on NTP  (Network Time Protocol)

Background – NTP will soon represent timestamps using two 64bit numbers; one for the whole number of seconds, and one for the fractional part of the second:

According to Mills, “The 64 bit value for the fraction is enough to resolve the amount of time it takes a photon to pass an electron at the speed of light. The 64 bit second value is enough to provide unambiguous time representation until the universe goes dim.”^[4] Indeed, 2⁻⁶⁴ seconds is about 54 zeptoseconds, and 2⁶⁴ seconds is about 585 billion years.

That ought to do it.

It’s a joke alright, but no April Fool

When I saw the timestamp on the tech press articles I hoped against hope. April 1st 9:43am.  Draft International Standard DIS 29500, more commonly known by the wonderfully double-speak name of Office Open XML (it’s neither Open, nor XML), is about to became IS 29500.

It would seem that Microsoft have been successful in fast-tracking their OOXML “spec” through the International Standards Organisation,  somehow managing to side-step the fact that their proposal is overblown, has no reference implementation, is not based on user need, and is full of bug-for-bug compatibility flaws with existing proprietary Microsoft Office products.

How they managed to get their provincial proposal past a committee containing 47 coutries that aren’t even able to use their own alphabets in OOXML URLs is anyone’s guess.  And all this despite the fact that an ISO standard for office documents already exists – Open Document Format.

One barrier to adoption still remains however, and it’s a big one.  The global market. A market containing large constituencies who are are less inclined to bow down to Microsoft. China, India, Brazil.  After the posturing of the Committee processes, this is where the real battle will be fought.

PostgreSQL 8.3 – on its way

PostgreSQL 8.3 is now in Beta2 and judging by the change log it’s going to be a good’n. There’s one new feature in particular which comes along at just the right time for me & phuser.com, plus some other very welcome enhancements.

 

More on those headlines features in a moment. The word is that v8.3 will be a performance release. Looking at the official change log, this is only evident by a list of fairly gritty technical changes – asynchronous commits, heap-only tuples, distributed checkpoints, lazy XID assignment, on-disk data size savings – you get the picture. It sounds cool (never a bad thing!), but the important thing is that all the hard work by developer community appears have really paid off. For example Jeff Davis has demonstrated very significant performance improvements over v8.2.5. (On an aside, recent SPEC tests showed that v8.2 was no slouch either, falling 15% short of Oracle 10g on a hardware costing almost 15% less ($65,500 Sun vs $74,000 HP hardware; $0 vs $110,000 software, respectively). Not content with this, the Man in the Hat says that the PostgreSQL Core Team plan to be faster than Oracle, possibly starting with v8.3.

 

So what of the new features in this release?

 

The Full-Text Search component is now considered mature enough and functional enough to finally make the move from contrib/ to core. This is nice – one less thing to worry about when configuring new platforms, and finally removes MySQL’s long-standing advantage of having (albeit inferior) FTS out of the box.

 

But the one v8.3 feature I’m really looking forward to playing with (and hopefully using in earnest!) is the new set of XML functions. Perhaps I’ll write about these in more detail when I have had a chance to try them out, but in short they should make it much easier to generate XML representations of data from SQL. One subset of functions in particular maps tables to XML, apparently making it very easy to dump tables and queries as XML. This is just what I need to implement Phuser’s upcoming export feature, allowing our users to get their Phuse data out in a consistent, portable format. It should be very cool!

 

Of course, we’re still in Beta. There will be the traditional series of additional betas and RCs before we get the official release. But now is the time to start reviewing the change log and seeing if you need to make any modifications to your apps in order to make the switch. Download a beta and do some testing. If you find anything fishy, report it to the Postgres devs – they need to know. Having said that, in three and a half years of intensive Phuser development, I have never once encountered a bug, flaw or crash in PostgreSQL, even in betas. It really is as rock-solid as they say, and a real tribute to the chaps who make it happen and their rigorous release engineering processes.

 

In defence of walls

Danah Boyd comes to the defence of the Walled Garden – the notion that certain parts of the web be protected from external gaze. It’s certainly a relief to hear someone question the prevailing orthodoxy (open==good, closed==bad).

To reiterate some of Danah’s ground, the very notion of a “wall” has somehow acquired an inherently negative connotation. This is a shame, because it is only by delimiting states and distinguishing between them that we create and perceive value. Value, in its widest sense, is the difference between two states, one state being more desirable than the other. My schoolboy physics reminds me that energy in any system naturally dissipates until everything reaches the same level – a mean steady-state – maximum entropy. So to create localised pockets of interest amidst the bland soup we must inject energy and impose structure. And that’s where walls come in.

Brave New Worldwideweb2.0

Amidst the clamour and frenzy that surrounds whichever star currently shines brightest in the web2.0 firmament (Twitter, Facebook, MySpace et al), it seems that some old-fashioned principles have fallen by the way-side.

Long before the WWW, in the dark years of WW2, our governments took pains to warn us of the perils of spilling our beans too readily. ‘Walls have ears’, we were reminded. ‘Loose lips sink ships’, the posters warned. Citizens were urged to be vigilant about what they might be giving away even in casual conversation. After all, piece together a few bits of seemingly innocuous information and before long a more telling picture emerges.

Sixty years on, and such cares are far from our minds of course. As each shiny new social networking site comes along, we’re happy to reveal all in exchange for a ticket to ride on The Next Big Thing. Once aboard, we do it all over again by giving it up for our 2479 new best ‘friends’.

Sanity-saving tip for database-upgrade-migration hell

As a web app in beta, phuser.com is still evolving rapidly. This means that I am often faced with the trauma of migrating a dataset from one version to another, sometimes with substantial changes in the database schema. I’ve developed various tricks for this – some are ugly but effective and involve things like sed, whilst others score more highly on the cool-o-meter.  Being the nice fellow that I am I will spare you the former and show you one of the latter.