Secure pull backups

[oddlama]

I was thinking about a more secure solution to automatically managing remote backups. Yet, I am not familiar enough with this codebase to judge whether my imagined approach would work out in practice.

The problem

With both push and pull configurations, you have to allow server A or B remote access to the other. This is not ideal. The "obvious" solution is using the 0-trust configuration, but that essentially requires a third server with access to both A and B. For manual backups this is perfect, but for automated backups this has some downsides (more required bandwidth, requires additional server).

Unprivileged pull configuration

Instead, I'd like to use a better version of the pull configuration with non-root access to the source server for backups. By using zfs allow, this unprivileged user is just given the permission to use the send subcommand and maybe hold if required. That way, the source server has no access to the backup server at all, and the backup server has only unprivileged access to the source server.

A classic pull or push approach always compromises both the backup and the source if the "privileged" side is hijacked. By using this unprivileged proxy-user, we can alleviate the risk significantly and protect the source servers in case the backup is compromised. The worst case scenarios I can think of right now:

• Source is hacked and the access to the unprivileged backup user is revoked.
• Backup is hacked and attacker gains RCE on all backup servers, but cannot modify live data.

Implications

Additionally, this requires separate snapshotting and pruning on all source servers, as the backup server cannot modify the source datasets. Therefore:

• Snapshotting is always a local timer/cronjob
• Syncs are pull-only. Syncing is initiated by the backup host and must be read-only on the source host (using zfs hold would be OK)
• Pruning on source hosts must be a separate timer/cronjob (can this interfere with syncing?)

(Aside: This idea is not compatible with push-configurations, because dataset creation requires the mount permission. Having the mount-permissions allows whoever pushes the data to compromise the target system unconditionally, e.g. by mounting a dataset at /etc)

I'd like to implement this approach on my servers but I am not sure whether zfs_autobackup requires any permissions I am not aware of. I'd be happy to receive feedback.

[psy0rz]

It sounds doable, just try it. :)

If zfs-autobackup gets some kind of permission error, you'll see it. Normally zfs-autobackup shows all the errors and raises exceptions if something fails.

Since its just using the zfs-command via ssh, i dont expect any weird problems.

You would need to run it with --no-snapshot and --no-thinning i think. And maybe --no-holds.

Edwin

[oddlama]

Thanks, that's great! Why do you think --no-holds could be required? I think it might be a good idea to allow holds.

Also I was wondering if zfs_autobackup acquires some kind of lock before snapshotting / thinning / syncing. What if a timer on the local system is creating snapshots, while a remotely initiated process tries to zfs send some backups? I figure if there isn't some kind of locking, zfs would act up due to the dataset being changed while sending an incremental stream.

[oddlama]

Hey there, small update on the matter: The concept turned out to work perfectly, and I'll put some details here for anyone interested. Essentially, this workflow properly separates local snapshotting and remote dataset replication, even for multiple backup sites. You will additionaly get proper privilege separation and the guarantee that the backup servers will not be able to access your data, if you have the source datasets encrypted with ZFS.

I primarily wanted to decouple source and backup snapshots. This means having all clients create their local snapshots independently, and have the backup machines handle their replicas independently. There basically will be a snapshotting service on each client that acts on its own, and each backup machine can pull any of those local snapshots. This also integrates perfectly with the way the hold system works in zfs_autobackup, so each backup machine has its own holds and you will never accidentally get in a desynced state.

To be prepared for multiple servers in different timezones, it might be a good idea to use a UTC snapshot time format. This way the snapshot times on your backup system stay consistent. (As of now, this feature is not yet released, but you can add --utc to all invocations once this is added upstream)

Setup

The backup machines (offsite1, offsite2, ...):

• Create an ssh key for root, which will be used to access the other machines.
• Create readonly dataset for each client to backup zfs create -o readonly=on backup_pool/clientXX.

For each machine to backup (clients):

• Create an unprivileged user (e.g. autobackup)
• Add the public key of each backup machine to ~autobackup/.ssh/authorized_keys.
• Allow the autobackup user to send datasets and manage holds: zfs allow hold,release,send rpool
• Mark all datasets that should be snapshotted locally with autobackup:snap=true.
• Additionally mark your datasets depending on where you want to have them replicated. So if rpool should be pulled by the backup machine offsite1, mark it with autobackup:offsite1=true. Repeat for all backup machines where you want that dataset.

The backup machines never create snapshots, to prevent having multiple snapshots at the same time for each backup machine. That's why we use autobackup:snap=true to specify that a dataset should be snapshotted. As an additional feature, we can have local-only snapshots by setting none of the offsite properties. But each dataset that should be replicated to some offsite backup must have the property autobackup:snap=true.

Systemd Timers (cron jobs)

Finally, all of the snapshotting and replication should be done routinely and automatically. I'd recommend using systemd timers for this, as they are resilient against missed jobs (power outage), and can be randomized in time a little (to avoid having all your 3 backup sites pull snapshots at the same time from your client machines). But of course you also use cron jobs for this task.

In this example, the two backup machines should replicate all of the client's local snapshots once per day.

zfs-local-snapshot.service

[Unit]
Description=Create and thin ZFS snapshots
Requires=zfs.target
After=zfs.target

[Service]
Type=oneshot
ExecStart=/usr/bin/zfs-autobackup \
        --verbose --exclude-received \
        --snapshot-format %%Y-%%m-%%dT%%H:%%M:%%SZ-auto \
        --keep-source 10,1h2d,1d2m,1w1y,3m2y \
        snap

zfs-local-snapshot.timer (runs every 15 minutes by default)

[Unit]
Description=Periodically run zfs_autobackup to create and thin snapshots

[Timer]
OnCalendar=*:0/15
Persistent=true

[Install]
WantedBy=timers.target

zfs-replicate-pull.service (example for offsite1, replace with offsite2 for other backup server)

[Unit]
Description=Replicate remote snapshots to local system
Requires=zfs.target
After=zfs.target

[Service]
Type=oneshot

# 1. Pull snapshots in group home from autobackup@client1.example1.com
# 2. Only thin our pulled snapshots, don't touch the source system
# You can have several ExecStart= options, one for each client.

# client1
ExecStart=/usr/bin/zfs-autobackup \
        --verbose --no-snapshot --exclude-received --clear-mountpoint --zfs-compressed \
        --snapshot-format %%Y-%%m-%%dT%%H:%%M:%%SZ-auto \
        --ssh-source autobackup@client1.example.com \
        --keep-source 9999999999 \
        --keep-target 1,1d2w,1w1y,1m2y,3m10y \
        offsite1 \
        backup_pool/client1
# client2
ExecStart=/usr/bin/zfs-autobackup \
        --verbose --no-snapshot --exclude-received --clear-mountpoint --zfs-compressed \
        --snapshot-format %%Y-%%m-%%dT%%H:%%M:%%SZ-auto \
        --ssh-source autobackup@client2.example.com \
        --keep-source 9999999999 \
        --keep-target 1,1d2w,1w1y,1m2y,3m10y \
        offsite1 \
        backup_pool/client2
# ...

zfs-replicate-pull.timer (replicate daily at 00:00, with up to 1 hour delay)

[Unit]
Description=Run zfs_autobackup daily to replicate remote snapshots to local system

[Timer]
OnCalendar=daily
RandomizedDelaySec=3600
Persistent=true

[Install]
WantedBy=timers.target

To monitor failed commands, you can add this to all .service files:

# Use this in the [Service] section on the above services to to start the `monitor-notify` service when the command fails.
# You hva to create that service manually and can send the output per mail, or to any logging service you require.
OnFailure=monitor-notify.service

---

Thank you for this awesome tool!

[sercba]

What about if you use --no-thinning instead --keep-source 99999999?

[oddlama]

What I want is thinning on the target dataset only

[oddlama]

So that the datasets that were backed up are thinned, but the source hosts (from which they are pulled) are not touched

[sercba]

I understand now, sorry. Thanks for your work.

[oddlama]

No worries!

[sohsmith]

Thank you for writing this up, not having to have root login via ssh enabled makes me feel much happier.
I've used a minor variation of this with the target doing the snapshots (so the autobackup user is allowed snapshot as well) and a source side cron job to thin the snapshots.
This is so much better than zrepl - even complex backup geometries are easy to set up.

[psy0rz]

@sohsmith very glad to hear this, thanks :)

[norsemangrey]

Great write-up! I am testing it on my system. I'm wondering about setting the base dataset for a client to read-only though zfs create -o readonly=on backup_pool/clientXX. When I do this I get an error ""failed to create mountpoint: Read-only file system" and I have to run the script again. I guess it would not be a permissions error as the script is running as root?

[oddlama]

It's been some time since I wrote the original thing. If I recall correctly the readonly base dataset should never be mounted in the first place, nothing should be stored in it. The backuped datasets will be newly created children of that one anyways, and should have their own permissions. The readonly was only a precaution, it will work without.

[norsemangrey]

Ah, ok, thanks for the clarification. Are you still using the original scripts or have you made any modifications to them?

[psy0rz]

this means the mountpoint doesnt exist yet, but it cant create a mountpoint since the parent dataset is readonly.

possible solutions: change the mountpoint. or make sure the mountpoint in the parent dataset exists in on the sourceside of the backup. (so it will be on the target side when you finished the next backup)

mountpoints and the creating thereof are a source of problems, also see #112

Ill try to improve upon this, or at least make it more clear whats happening.

[norsemangrey]

The solution to this was simple. I just needed to unmount the base (read-only) dataset I created before running the script. There is no reason for this to be mounted anyway.

[psy0rz]

right, thats also a posibility.