download my resumeashley@mervyngraham.com

Ashley Mervyn Graham

rsnapshot Over a Network

Scenario

So. You have several terabytes of data that need to be backed up, not just once, but revolving. Okay, no problem. Let’s use rsnapshot. We could use rsync all by itself; but rsnapshot does a lot of low-level file-management this kind of setup would need for us, or rdiff-backup which does take up less space, but can be cumbersome when you have to retrieve a file.

So We’re Settled: rsnapshot It Is

I’m going to skip past installation, because well, if you can’t install it you probably shouldn’t be using it.

Well maybe not skip. Just extremely glossed over. If you have a package manager (like one of these: apt, rpm) on your Linux install; look there.

We are going to keep this recipe simple: one server backing up itself and another client over SSH, and as such this is the focus, setting up rsnapshot will also be glossed.

So, you have rsnapshot installed on the server, good. First things first though, make a backup of that configuration, so later, when you’re fiddling and brake something, you can revert back to a known good copy.

I’m going to be just-as-glossy as before when I describe rsnapshot and it’s default settings, you really should read the HOWTO for rsnapshot and the included rsnapshot.conf file in your rsnapshot installation.

Of course if you know what you’re doing, followed these steps before and want to match my setup entirely, my rsnapshot.conf file is available for your (or mine after a hardware error) perusal.

rsnapshot Configuration

There are a few small points of interest for this to all work, notably ssh has to be enabled so we change this:

# Uncomment this to enable remote ssh backups over rsync.
#cmd_ssh        /path/to/ssh

To this:

# Uncomment this to enable remote ssh backups over rsync.
cmd_ssh        /path/to/ssh

A quick whereis ssh from the command line should tell you where SSH is located.

Then we have to enable batchmode for SSH:

# ssh has no args passed by default, but you can specify some here.
#ssh_args       -p 22

Should become:

# ssh has no args passed by default, but you can specify some here.
ssh_args        -p 7654 -o BatchMode=yes -i /home/rbackup/.ssh/id_rsa

Note that I use a custom port (the -p) and define the identity file to use (the -i). Also, we generate this identity file a little bit further down the page, don’t worry if you don’t have it yet.

The last change required is to tell rsnapshot to use the user (we’re going to create in a little bit) when connecting to the client:

backup          rbackup@client:/path/to/backup/    backup_dir/

This should be placed along with all your other backup scripts/points at the end of /etc/rsnapshot.conf.

Remote Backups Security Considerations

rsnapshot should be ran as the root user so it can access everything on the file-system without restriction, which means that when we have it access clients it too should be ran as the root user, which is a generally bad idea. Instead we’re going to fake it, with a non-privileged user who can use rsnapshot as root under special circumstances. It’s safer than doing all of this as root but can still be exploited so be careful.

Meet rbackup

We’re going to need a user on both the client and the server for this. On the server, root is going to pretend to be them when using SSH onto the client, and because I can’t seem to get root‘s public RSA key to work, he’s going to hijack another user’s identity in the process, to keep it all clean, it just so happens to be the same user as the client.

In this example, the user is rbackup; it needs to exist on both machines, so add a user on both machines. The name you choose doesn’t matter, just make it the same for ease.

It’s a good idea now to become rbackup for a bit and setup passwordless SSH for rbackup.

Lockdown The Client

We’ve given rbackup the ability to connect to our client from our server, we need to make rbackup capable of executing a command as root on our client. We don’t want rbackup to have admin rights to do anything, just one thing: run rsync as root so it has access to everything we need it to.

We need to edit the sudo file on the client, it’s very important you use the visudo command, otherwise you could break something and be forced to enter recovery mode to fix it.

sudo visudo

After using the above command, you might be presented with a warning about a syntax error. Pressing your "e" key will allow you to re-edit the file.

And add this to the bottom:

rbackup ALL=NOPASSWD: /usr/bin/rsync_real

You can do some research on what this means but essentially we’re allowing rbackup to run /usr/bin/rsync_real as root without a password from any terminal (that’s the ALL).

/usr/bin/rsync_real doesn’t exist, it is going to be called from the new version of rsync we’re going to create. We have to do this because when we connect with SSH later we’re going to have to call sudo rsync to get the privilege we need, however we can’t call sudo rsync, we can only call rsync so we’re going to change /usr/bin/rsync to suit our needs.

First, move the original rsync out of the way:

sudo cp /usr/bin/rsync /usr/bin/rsync_real

Then create a new /usr/bin/rsync with your editor of choice: ne /usr/bin/rsync:

#!/bin/sh
/usr/bin/sudo /usr/bin/rsync_real "$@"

Be careful if you’re merely going to copy-and-paste that, the quotation marks can become irregular characters. Unless you’re sure it worked, do yourself a favor and delete the quotation marks and replace them manually.

Usually when we run rsnapshot on our server it tries to connect to our client with SSH and execute rsync as rbackup which doesn’t have access to anyones files.

However now that we’ve changed rsync and added rsync_real, when rsync gets called we’re actually running sudo rsync, and because of our change to our sudoers file, rbackup is authorized to run rsync_real as root without a password, allowing it to access anyones files.

Lockdown the Connection

We’ve already setup the keys for rbackup to have access, but we can do better, we can limit what rbackup can run and where it can connect from.

On the client, take a look at /home/rbackup/.ssh/authorized_keys it should contain one entry that looks somewhat like this:

ssh-rsa AAAAB3NzaC1kc3...77dhKbnd77sq11HWIA= rbackup@server

Imagine the three dot’s are replaced with more random alpha-numerical characters.

We can add some magic to this key to limit where it can connect from, and what it can do.

First, if we know what the server’s IP address, or hostname is we should limit it to that:

from="192.168.1.100"

There are a lot of options to limit connecting hosts, as well as leaving it out entirely.

We can also force the connection to execute a single command:

command="/home/rbackup/validate-rsync.sh"

So at the end your authorized_keys file for rbackup on the client should look like this:

from="192.168.1.100",command="/home/rbackup/validate-rsync.sh" ssh-rsa AAAAB3NzaC1kc3...77dhKbnd77sq11HWIA= rbackup@server

Not that there is not a space between from="192.168.1.100" and command="/home/rbackup/.ssh/authorized_keys", any limits you place on a key are separated by commas.

If you’re unsure about where rsnapshot will be connecting from it should look like this:

command="/home/rbackup/validate-rsync.sh" ssh-rsa AAAAB3NzaC1kc3...77dhKbnd77sq11HWIA= rbackup@server

Next, we need to produce a /home/rbackup/validate-rsync.sh file on the client:

#!/bin/sh
case "$SSH_ORIGINAL_COMMAND" in
*\&*)
echo "Rejected"
;;
*\(*)
echo "Rejected"
;;
*\{*)
echo "Rejected"
;;
*\;*)
echo "Rejected"
;;
*\<*)
echo "Rejected"
;;
*\`*)
echo "Rejected"
;;
*\|*)
echo "Rejected"
;;
rsync\ --server*)
$SSH_ORIGINAL_COMMAND
;;
*)
echo "Rejected"
;;
esac

Make sure the file is executable by rbackup: chmod u+x /home/rbackup/validate-rsync.sh and that rbackup owns the file: chown rbackup validate-rsync.sh

Now, only a connection that matches above will be passed through, everything else will be rejected.

Testing

For this test we’re going to remotely backup a single file (so we can get instant results to see if SSH is working properly). Comment out all of your backup points in /etc/rsnapshot.conf and replace with a single remote one:

backup          rbackup@client:/path/to/backup/file    backup_dir/

Make sure this file exists first.

Then run from the server: execute the command sudo rsnapshot hourly (replace "hourly" with whatever your first interval is called).

With a little luck–and depending on your verbosity setting (the default is quiet, save for errors)–after a little while you should be presented with your terminal prompt again, meaning the command executed without any hiccups. Great.

However, errors can happen, any errors should be investigated. But how?

The most general issue is a problem with your key; the problems are too far and wide to begin describing here: google will be your best friend.

To aide in debugging the problem, you might want to consider turning on verbosity for SSH. Change your ssh_args in /etc/rsnapshot.conf file to this:

ssh_args        -vvvp 7654 -o BatchMode=yes -i /home/rbackup/.ssh/id_rsa

The difference here is the addition of the three v‘s, at the beginning of the flags, this turn on the maximum level of verbosity.

If you want a lot more verbosity, try turning on verbosity on the client too, edit /etc/ssh/sshd_config on the client and change this:

# Logging
SyslogFacility AUTH
LogLevel INFO

To this:

# Logging
SyslogFacility AUTH
LogLevel DEBUG3

On a shared SSH server (one with other people) this setting will break their privacy, don’t leave this setting on. Be sure to change it back when you’re done.

Then restart the SSH server on the client: sudo /etc/init.d/ssh restart and give it another shot from the server: sudo rsnapshot hourly. Hopefully the output and a bit of google should aide you in your woes.

Since the various problems are too vaired, and I can’t even begin to jot them down all here (along with solutions), we’re going to move on.

Automation

Essentially, all you need to do is add an entry to root‘s cron file: sudo crontab -e, I have an example all setup already.