Rustic support in Backrest

[garethgeorge]

Hey, backrest developer here! Backrest is a multiplatform WebUI for restic. Discovered the rustic project some months ago and as it matures I'm interested to learn more about it / whether it's a good option to support in the WebUI.

Reaching out to learn more about the state of the project and whether rustic is a backend that I might support in the backrest project. To support a new backend it generally needs to support the operations in restic.go in the backrest project.

A quick search of the repo shows that there is some amount of support for JSON output in the code base (which is awesome to see!). Backrest works by forking restic and parsing it's JSON mode output for many commands. I'm interested to hear how close to restic rustic is and what major differences there might be that I'll run into. I'm thinking compatibility may be very easy if there's generally a 1:1 correspondence between the commands and generally the data available in the JSON output from commands (even if the fields are difference this can be addressed by converters).

I notice that rustic also offers a library -- linking against rustic might be an option here too! Questions I have here would mostly be around how I can best capture and surface any rustic logs etc if calling core.

[aawsome]

Hi @garethgeorge!
Thanks for you interest in rustic! In principle, rustic is not 100% CLI compatible to restic, but very similar. Most commands work like restic or at least very similar.
So, if you want to use rustic by calling the CLI, I would say just try it out - it will work pretty much like restic.

There are a few things to note however, which are mainly due to the differences of restic and rustic, see https://rustic.cli.rs/docs/comparison-restic.html:

• Configuring remote repositories is quite different as rustic uses opendal for this. On the other side, more backends are supported directly compared to restic
• If you want to use hot/cold repositories, this also needs extra configuration and commands may work differently.
• Some commands do not exist while others have additional functionality
• Note that rustic is lock-free

A better option would be to use the rustic_core crate where you have much finer control over the repository commands and direct output, see https://docs.rs/rustic_core/latest/rustic_core/. But I fear the main point here would be to make it accessible to a Golang project. Besides this, progress information is basically available by setting a ProgressBar to the Repository which is a callback-function. logs are created using the log crate, i.e. you can define your own log handler (again basically a call-back) and do whatever you want.

[aawsome]

For your BackupRequest, BackupOptions is also seralizable and includes parent related options.

I suppose one interesting question to ask is -- as I'm thinking of heading off in this daemon direction, does this make sense to you?

This actually makes perfectly sense to me. You just have to make sure that sometimes you have to re-open the Repository instead of using the existing one. For each backup request, you should do

let repo = Repository::new(&repo_opts, &backends)?
        .open()?
        .to_indexed_ids()?;

before calling repo.backup(), for example. And you can directly throw away that repo after that call.
This mimics the CLI-oriented design rustic_core still has. But it will be still more efficient and flexible than a CLI-call would be. Also the libraries would all be compiled into your daemon binary with no need of an additional binary. Plus, we may soon see some repo.update() in rustic_core which will be much more efficient and will allows you to adapt to with only small code changes.

Moreover, your daemon could allow for navigating around dirs in a snapshot. Here, the repo can be kept and re-used for each user action.

[aawsome]

Followinp up re: wrapping at the CLI level, I think the only capabilities that are strictly required for an MVP are

* ability to parse the ID of the last created snapshot out of the backup command (and ideally parsable progress though I could simply show the TUI text in the UI).

* parseable output from snapshot listings

As a followup I'd also want

* parseable output from forget

* parseable output from operations for browsing snapshots

For the rustic CLI, you don't have parseable progress and to parse rustic ls you have to parse the text output. All other requirements are fulfilled. (And you get all when using rustic_core)

[aawsome]

About efficient updating of snapshots, there is now rustic-rs/rustic_core#260.
For efficient index updating, see rustic-rs/rustic_core#261

[garethgeorge]

Thanks for those updates! Spent a bit more time experimenting this weekend

Now that I've had some time to prototype out the gRPC approach (which is fairly minimal so far but gives me a feel for what I'd need to maintain in terms of a daemon process) garethgeorge/backrest#462 I'm becoming torn on the right way to proceed, I see some cons with reinventing the wheel with the gRPC daemon

• I think a lot of users like Backrest because it's such a thin wrapper e.g. if they're familiar with restic, that's also the behavior they get from Backrest. Not sure whether it makes sense for Backrest to be in the business of building an opinionated engine that diverges from the base project.
• Build system complexity: I'll need to manage and ship releases of the daemon from some project. Right now I'm building it in a mono-repo but it probably needs to be a separate project with it's own pipelines and release schedule.
• Configuration divergence -- rustic supports a number of flags and env vars in its CLI which are well designed and closely mirror the options restic exposes, I want to keep that familiarity / pass it on to users and am realizing anything I do here will duplicate work you're already doing on the CLI.
• There isn't a direct pathway to support the "Run Command" feature which lets users craft their own CLI invocations of "restic" today or "rustic" in rustic mode.

The obvious pro of the daemon / building on rustic_core is the level of control but it's arguable that Backrest is trying to be closer to a scripting layer than an actual backup engine!

I pivoted a bit to spend some time playing around with the rustic CLI itself and I'd somehow entirely missed the JSON support in my first look at the project. If I switch rustic to log to a file (logs sometimes interrupt JSON output) it looks likely to be straightforward to parse and the output you generate is already relatively possible to transform into the structs I use for restic.

Wrinkles I see with automating by wrapping the CLI are pretty slight:

• Lack of JSON progress is a bit tricky but I'm imagining JSON progress bars are something I could PR for a future release I suspect
• Logs occasionally print while producing JSON output -- will likely be addressed by just logging to a file instead of stdout.

Otherwise CLI behavior is very close to restic today and I can likely support both under a single interface. Aiming to find some time this week to prototype this as if it's as easy as it's looking I might be able to get this up and running in a few days.

So current status is -- likely I'll continue a bit further with the gRPC approach but I'm also going to look at just wrapping the CLI and see how stable I can make that. If it is stable this is probably my preference barring upstreaming a gRPC daemon to rustic. WDYT?

[aawsome]

Thanks for keeping us up-to-date @garethgeorge
I can understand your considerations. And of course you can use the rustic CLI itself and it will be able to replace restic in most circumstances.

About JSON progress: This can IMO be implemented pretty easily by using another ProgressBars implementation when --json is specified. If you want to work on this, I can support you.

About logs: you can log to a file or use the --log-level option: You may want to set it to error or warn. But also note that logs go to stderr (like progress information) whereas json output goes to stdout. If you take this into account you may just use rustic as-is - maybe just directly show stderr to the user and read/interpret stdout json.

[simonsan]

One thing hasn't been mentioned here, there are two libraries you can use. One is rustic_core which is the low level library. The other one is rustic-rs which is the cli library - you find it in the main repo under /src. Main.rs is the binary (a very thin wrapper around the lib) and everything else is the library. We haven't used it for that (because we really just call into the apps entrypoint) but we probably can make it possible to call the high level commands in code. Maybe you can look into that approach and see if you need any changes there to make your use case possible. Most Runnable commands can be also called from outside of the trait implementation via run_inner/inner_run. I would recommend using your own branch of that library and PR the changes you needed to make to make it work, a lot of items you will need are currently not visible/pub. You can change that and we discuss about it and what other changes are needed. I might invest a bit of time after the weekend to decouple some logic there from the commands. So they become easier to use without abscissa and can be easier tested as well.