[Question] Incremental snapshot backups and usage

[sww1235]

Hello: I am running into an issue with incremental snapshots and am wondering if I understand their usage correctly.

I am running FreeBSD 12.2 Release P1 with the 0.4RC FreeBSD binary

/usr/bin/zfsbackup version
       Program Name:   zfsbackup
       Version:        v0.4
       OS Target:      freebsd
       Arch Target:    amd64
       Compiled With:  gc
       Go Version:     go1.14.2

I am using periodic-snapshot to create hourly snapshots of 4 zfs pools, and then running the following commands in a script in /etc/periodic/daily/ to run incremental backups. (These lines are duplicated 4 times for each of the 4 pools)

/usr/bin/zfsbackup list --encryptTo user@domain --signFrom user@domain --publicKeyRingPath /home/user/.gnupg/public.pgp --secretKeyRingPath /home/user/.gnupg/private.pgp b2://the-vault-remote/backups"

/usr/local/bin/flock -xn /root/zfsbackup-backups.lock -c "/usr/bin/zfsbackup send --encryptTo user@domain --signFrom user@domain --publicKeyRingPath /home/user/.gnupg/public.pgp --secretKeyRingPath /home/user/.gnupg/private.pgp --increment the-vault/backups b2://the-vault-remote/backups"

I used the same commands, but with --full instead of --increment to perform an initial backup of the pools, but when the incremental script runs, I get the error message:

Found 0 backup sets:

2021/03/13 03:03:39 Selected incremental snapshot does not exist!

The initial full backup commands were run as root, and /usr/sbin/periodic is being run as root in /etc/crontab.

Any help would be appreciated.

[someone1]

Hmm - is there a reason why you're doing a list before the actual backup?

Can you provide the log output with --logLevel debug set?

[sww1235]

I was running list before the actual backup as a debugging tool, was trying to see if the backups were being detected at all.

I added the --logLevel debug option into my /etc/periodic/daily script so by tomorrow, I should have the logs for you.

Thanks for the help.

[sww1235]

daily.log

I attached the stripped down output from /var/log/daily.log. Didn't think you wanted to see that my passwd file was backed up.

[someone1]

If you go to backblaze and look at the contents of the-vault-remote, what do you see? Can you list the path of any .manifest file you find?

[sww1235]

so under the main bucket the-vault-remote, directories:

• backupsmanifests|the-vault
  • backups|daily.0.manifest.gz.pgp
• backupsthe-vault
  • backups|daily.0.zstream.gz.pgp.vol1
• mediamanifests|the-vault
  • media|daily.0.manifest.gz.pgp
• mediathe-vault
  • media|daily.0.zstream.gz.pgp.vol1
  • ... (472 files omitted)
  • media|daily.0.zstream.gz.pgp.vol474

[someone1]

Apologies - the notification of your response got buried!

I wonder if this is some silly prefix issue... does using b2://the-vault-remote/backups/ instead of b2://the-vault-remote/backups fix the issue?

Apologies that I can't help more at the moment, been trying to build a new dev box as my current one is unusable - been months waiting on part availability due to shortages.

[sww1235]

No worries! Will check and report back.

[sww1235]

@someone1
After wiping the bucket and reuploading the full backup using the slashes at the end of the path, I am still getting 0 backups found in the zfsbackup list output, and no incremental backups found on the zfsbackup send.

See attached log file from a manual run of the script

periodic-daily-output.txt

After adding the slashes to the end of the backend path, it did change how the folder was laid out in backblaze but nothing else.

Now I get

• the-vault-remote
  • media
    • manifests|the-vault
      • media|daily.0.manifest.gz.pgp
    • the-vault
      • media|daily.0.zstream.gz.pgp.vol*

Thanks for all the help with this.

[sww1235]

@someone1 have you had a chance to look at this yet?

[someone1]

Apologies - I think I need to add additional debug logs to the program as I'm not seeing anything obvious, but I am missing details as to what its trying to work with.

I'll have to sign up for a backblaze account again and run some tests. Maybe you can test and see if the S3 compatible API will work for backblaze? Try setting AWS_S3_CUSTOM_ENDPOINT=https://s3.<region>.backblazeb2.com. You can read more about the setup here (full disclosure: I haven't tried, let alone validated this).

I appreciate your patience on the matter.

[sww1235]

If you want me to run any tests, let me know. I am also starting to dig into your code a bit and add some debug logging myself.

I will try their s3 api as well

[neuralsandwich]

I've been hit with this as well. Strange enough though, on smaller tests I did everything worked out fine 🤔. I could take a look at this in a few weeks if nothing found before then.

The S3 backend also doesn't appear to work either. I suspend it could be the original snapshot no longer being available on disk but I'm quite new to zfs