Make Vorta profiles easy to backup and restore

[patogit]

First -- thank you very very much for making such great free/libre software! I've used GNU/Linux for about ten years, and whenever I looked at backup solutions, they all seemed too simple or way too complex. Vorta gives me secure backups with detailed control, sane defaults and an easy to understand interface. A delightful creation!

Summary:
I want Vorta to let me save profiles and then load them on a different system.

Details:
I just used Vorta to backup my system and user files, then I erased my disk. Now I've installed a fresh operating system and installed Vorta. I open Vorta, and want the option to load my old profiles, but that option doesn't exist. So I create a new profile and "Add Existing Repository" and give it the location and passphrase, and it successfully loads the archives. But now, if I want to make a new backup, I have to recreate my old profile! In this case, the most tedious and error-prone part is the Sources tab -- can I remember exactly what sources and exclude patterns I used to create these backups? If I can, great. If not, time for another round of trial and error and honing the details.

Where are my profiles? I looked for the profiles, both on my new system and the mounted backup, and I don't see them anywhere. I see ~/.config/borg/security has the same number of directories as I had profiles, but the files don't have my sources list. If I could find my profiles, that would be a first step towards backing them up and loading them. As it is, Vorta appears to function in a way that I can't back up my backup config :). This means that I can't integrate it with config managers either, like nix, guix, or aconfmgr.

[Hofer-Julian]

@patogit thanks a lot for the nice words :)

I agree that it would be helpful if exporting/importing your profiles would be easier.
If you know Python, a PR would be very appreciated.

Until this can be done in the GUI, you can find your settings in:

• ~/.var/app/com.borgbase.Vorta/data/Vorta/settings.db if you use flatpak
• ~/.local/share/Vorta/settings.db if you use pip

[patogit]

Thanks for the prompt response @Hofer-Julian. I don't know Python, or much other coding, but I can poke around and tweak things. I found the settings.db where you said, put it in place on my new system, and it works! I see that it brings all my profiles. So, migrating all my settings and profiles at once is easy (by manually moving that file), but exporting and importing a single profile requires database work that I don't know how to do. For now, my immediate problem is solved: I have my profiles. Moving forward, I see two projects:

• Add documentation explaining how to migrate settings and profiles by moving settings.db. If you suggest an appropriate place for this, I could do it.
• Code a GUI way to export and import a profile or set of profiles. I would need more guidance on this.

Maybe this brings up security concerns? I doubt it, but hey, each file floating around telling someone about my backups offers them insight into my system. This is each users responsibility not to leave lying around, but maybe something can be done to minimize risk.

[m3nu]

For a proper export, we also need to deal with saved repo passwords. Those are not always in the DB, but use the system keychain if one is available.

[patogit]

Idea: The settings export dialog can offer fine grained control of what to export, with a column for each profile and a row for each category of settings (repo, sources, schedule/logs/commands, archives/prune options). Also a space at the top for the Misc settings that affect how Vorta runs. Each of these would be a check box, so that I can select, for example, all the settings of my first profile, and only the sources settings of my second profile, and export all of that into a file.

[patogit]

Details of how I scared myself this morning:
This was going to be titled "Archive mounted at system shutdown has disappeared"

1. Made a backup on May 14, and another backup on May 15, both times to a repo on an external hard drive to an ext4 partition.
2. Erase system, fresh install of Manjaro and Vorta.
3. Configure Vorta enough to mount the May 15 backup and get my old settings.db.
4. Quit Vorta, put my old settings.db in place, restart Vorta.
5. Mount May 15 archive again, access some files.
6. Shutdown my computer without unmounting the archive.
7. Turn my computer on, open Vorta, and the May 15 archive is gone.
8. Mount May 14 archive, get old settings.db and notice that it now says it was last modified May 14.
9. Quit Vorta, put my old settings.db in place, restart Vorta.
10. Now the May 14 archive is gone from the list in Vorta! I'm not going to mount the previous archive or do anything else until I can figure out where the May 14 and 15 archives have gone!!!!

I think I understand at least one piece of the puzzle: an archive that includes settings.db does not include a reference to itself, because settings.db adds the new archive after the backup completes (of course). So each step I take back in time to get a settings.db takes me back another step in time. Soooooo..... in order to get out of this cycle, I fetch the old settings.db but I don't use it directly -- rather:

1. Open old settings.db with an SQLite browser and copy as plain text the data I want (in my case, the Sources include and exclude lists).
2. Open Vorta with a blank settings.db, create a new profile and add my existing repo.
3. Ta-da! All of my archives appear!
4. Paste my old sources lists into my new profile.
5. Breathe a sigh of relief that nothing was lost, it was all in my mind.

[SommerEngineering]

Thanks @Hofer-Julian for the paths. In my case (Ubuntu 20.04 + Flatpak) it was: ~/.var/app/com.borgbase.Vorta/data/Vorta/settings.db. May this helps others as well.

[Hofer-Julian]

Thanks @Hofer-Julian for the paths. In my case (Ubuntu 20.04 + Flatpak) it was: ~/.var/app/com.borgbase.Vorta/data/Vorta/settings.db. May this helps others as well.

Absolutely correct, it seems like I forgot the ~/.var prefix, I've edited my comment.

[amarendra]

Or have the settings at a fixed location something like ~/.config/vorta/settings.db across platforms so that one can symlink the settings from elsewhere.

And/or for read a borg patterns file.

[m3nu]

Or have the settings at a fixed location something like ~/.config/vorta/settings.db across platforms so that one can symlink the settings from elsewhere.

That won't work, since we need to follow the platforms conventions on file locations.

BUT we can add a link to the folder, like we already do for the log folder.

[ghost]

@patogit thanks a lot for the nice words :)

I agree that it would be helpful if exporting/importing your profiles would be easier.
If you know Python, a PR would be very appreciated.

Until this can be done in the GUI, you can find your settings in:

• ~/.var/app/com.borgbase.Vorta/data/Vorta/settings.db if you use flatpak
• ~/.local/share/Vorta/settings.db if you use pip

it's here on a Mac:
~Library/Application Support/Vorta/settings.db

[samuel-w]

I found this gist https://gist.github.com/zdxerr/8491076. Unclear what the license for it is, but it should be useful.

[samuel-w]

Would an acceptable solution be copying and restoring the entire database, rather than individual profiles?

[m3nu]

We don't need code to copy the database. Just a link to the settings folder or some details in the docs would be enough.

One use case for import/export is automated deployments to e.g. company laptops via some kind of MDM. For this we need some kind of export format.

There was also the suggestion to import/support Borgmatic settings (YAML format).

[samuel-w]

Peewee does have model_to_dict and dict_to_model methods. I could use that, and this logic for imports:

• Overwrite settingsmodel on import based on checkbox
• Check for conflicts of:
  id, repoid, repourl
• If id collison increment id , if repourl collision set repomodel to preexisting repomodel
• if password is not in system keyring and encryption != 'none':
  prompt for password entry

For exports just write the output of model_to_dict to a file.

You would only need to backup the password if its in the vorta database, since the other system keyrings have files that can be backed up individually. The profile backup feature would most likely be used on similar systems that both have the same setup of system keyring, such as laptop to PC.

Oh, and maybe there could be a setting of including the profile backup in the borg repository.

[samuel-w]

Where should I put the profile backup/restore button?
I could put it in:

1. A button beside all the other buttons
2. Inside the selector similar to the repository add/initalize functions
3. The misc tab
4. Somewhere else

[m3nu]

Have we already decided on all other details? Export format? What to export?

Regarding where: Maybe near profiles in a sub-menu, if the export is based on profiles. We'll have too many buttons (that aren't used often) after adding import/export, so I suggest a new sub-menu with [...] like "Other Profile Actions". "Add Profile" would still be at the top-level, but the rest can be hidden, since those aren't used so often.

[samuel-w]

Its easiest to export to json, as there are model_to_dict and json.dumps(dict) methods.
For exports, everything could be exported, and then on importing what to import would be selected.

I think a single checkbox for importing SettingsModel would be all is needed, since the typical use case would be setting up a new installation quickly, as stated in the first post.

For the buttons, maybe there could be an button to expand/collapse the other buttons, like this:


with a tooltip of "More buttons" and "Less buttons"

The always visible button is "Add Profile"

[m3nu]

Collapsing buttons is not a very common UI element on desktop. It's more common to use a submenu. E.g. her in the network settings.

[samuel-w]

Ah, OK. Menus have to be generated using code, there is no way to do it in the Qt Creator, right?

[m3nu]

You could do them as QML, but probably simpler in code.

[samuel-w]

I have created a backup/restore system, but I need someone that has wifi settings on a mac to test it . https://github.com/samuel-w/vorta-1/tree/BackupRestore

[m3nu]

Will test it. Thanks.

[samuel-w]

How should schema upgrades be handled?

[m3nu]

We have versioning and migration code in models.py. For the export I'd focus on the very basics to start with. Those don't change generally. For everything else the default value can be used.

[samuel-w]

For upgrading, we could:

1. Create a new temporary database of just the profile being restored
2. Migrate the temporary database
3. Merge the temp database into the current database

This would require seperating the migration logic from init_db. Importing the profile and then migrating would not work, as there could be columns in the profile that do not exist in the database.

[m3nu]

Can you open a pull request when the changes are ready? Easier to see the changes that way.

Importing an older version can be complex and changing the DB is risky. Better to only import all known options and just ignore the rest.

[claudiorrrr]

I was just checking in on this issue and I realized that in my case the settings.db file is located at ~/Application Support/Vorta

In case that's useful for others.

[phihos]

Any update on this? This feature would be extremely useful for the company I work in. Can I help somehow?

[m3nu]

There is some incomplete code that was written by a former contributor, but it needs more work.

Since we use Peewee internally, there are some existing helper functions: https://stackoverflow.com/a/21979166/3983708

Your use case would be to export one profile, rather than the full DB? (you can already copy the full DB to another machine to transfer all settings).

If you're still interested, I could add a new PR for this feature with the code that exists and you help polish it up for your own and general use?

[phihos]

Creating a new PR for me would be great 👍
My use case is to provide the initial configuration via config management for many desktop clients. Since each client has a different user and SSH key simply copying a preprovisioned database file won't cut it. It would be great if I could template a JSON file, put it in a specific location and it gets picked up by vorta when starting.
So in short my use-case is to artificially manufacture a backup and auto restore it to provide a scalable borg backup deployment via config management.

[m3nu]

I see. Someone had a similar scenario last year. I’ll see what code is around for this feature. Give me until tomorrow or so.

[phihos]

Thanks, looking forward to it :-)

[m3nu]

Here the draft PR for this feature, @phihos. #954

Still needs some work. I added my comments there.