Improve CLI's support for scripting and automation

[divoxx]

Mutagen CLI is very user friendly but not so much when trying to automate. Specifically, the output of the commands are not easy to parse.

It would be nice to provide an option to control the output. Some examples and suggestions:

mutagen create

Currently,

$ mutagen create /foo /bar
Created session 9dd2b35f-48a6-4c63-891e-73c8065703c0

If mutagen is being called from a program, it's unnecessarily hard to get the session ID. Instead, it would be nice to have a way to simply print the ID.

$ mutagen create /foo /bar
9dd2b35f-48a6-4c63-891e-73c8065703c0

mutagen list

Currently,

$ mutagen list
--------------------------------------------------------------------------------
Session: 9dd2b35f-48a6-4c63-891e-73c8065703c0
Alpha:
        URL: /foo
        Status: Connected
Beta:
        URL: /bar
        Status: Connected
Status: Watching for changes
Conflicts:
        (α) Dockerfile (<non-existent> -> File (31dc7a94e4600804e2a79caa0de95fa00f2c6df0))
        (β) Dockerfile (<non-existent> -> File (98dfebc4eaeca05f5b0c599d6deedec176acc5af))
-------------------

This is very hard to parse, it would be really nice to have a machine friendly output. Something like a JSON output would be ideal.

$ mutagen list --json

{
  "session": "9dd2b35f-48a6-4c63-891e-73c8065703c0",
  "alpha": {
    "url": "/foo",
    "status": "connected"
  },
  "beta": {
    "url": "/bar",
    "status": "connected"
  },
  "status": "Waiting for changes",
  "conflicts": {
    "Dockerfile": {
      "alpha": "<non-existent> -> File (31dc7a94e4600804e2a79caa0de95fa00f2c6df0)",
      "bera": "<non-existent> -> File (98dfebc4eaeca05f5b0c599d6deedec176acc5af)"
    ",
  },
}

others?

There are possible other commands that could be improved but those two would make it a lot simpler to interact with mutagen from scripts.

---

Is this something you're willing to add/change? I would be happy to try and help if you give the 👍 .

[xenoscopic]

I agree this is sorely needed. I'm desperately trying to get back to Mutagen development to work on it, because there's a lot more that I want to do, and I know people are waiting. It's highly likely that I'll be hammering out new code for Mutagen as of Thursday, maybe even tomorrow.

There was some previous discussion on this particular issue in #48.

The JSON output option is an interesting idea. The Protocol Buffers messages used for the daemon gRPC API already define a canonical JSON serialization format, though I'm not sure it's very pleasant or idiomatic. It wouldn't be hard to make JSON work somehow though. I'd also like to prototype a simple line-based format for piping into shell scripts that don't have JSON parsing readily available, but there's no reason that there couldn't be a --format=<default|json|simple|...> flag for the list and monitor commands.

Also, as described in #51, I think some thought needs to be given to the many-conflict scenario and what to do with that, especially if doing streaming monitoring, but the solution will probably be the same for JSON and other formats.

As people start to bring in contributions (which are all very much appreciated), I'd like to set up some contributing guidelines, as well as a CLA or similar. I have some ideas for other projects that I want to build on top of Mutagen and I want to make sure that Mutagen itself stays open-source and free of any licensing problems for me and everyone else. I honestly wasn't expecting contributions so soon, so I wasn't really prepared for it. I'm working on it though!

[divoxx]

No rush, working on open source is time consuming and more often than not they are side projects. Take your time.

I'm creating the tickets based on pain points I've been feeling as working with the tool but they are not urgent since most of the limitations can be worked around.

In terms of contribution, having guidelines would be helpful. I'd be willing to contribute but I don't want to spend the time going in the wrong direction, etc.

About this specific JSON, I think the gRPC JSON serialization would be good enough for most cases and a great starting point. Parsing JSON from shell/CLI is not hard with tools like jq

[justjake]

@havoc-io in the distant future, it would be great to support a conflict resolution API for mutagen. The current strategy of "await FS resolution" works great for interactive sessions, but for anyone wrapping mutagen in other tools, it passes the burden of remote-control onto those users.

Imagine being notified of a conflict in an IDE that wraps mutagen via mutagen list --format=json: if the IDE wants to force preferring the client-side changes, it also needs to implement ssh $BETA_HOST rm $BETA_PATH/$CONFLICT_PATH and all the failure case handling around that.

It would be easier for a consumer to wrap a command like mutagen resolve $BETA_PATH --op=delete.
I just wanted to add this to the conversation 💁

[xenoscopic]

@justjake I'm hoping that the new synchronization modes described here (in particular two-way-resolved) will eliminate some of the need for interactive resolution, though in the long run I still see the utility in having a mutagen resolve command to manually resolve conflicts without the need to an interactive session (especially if we're talking about scripting from other tools).

I've actually prototyped this a bit, but the main issue I ran into was how it interacted with synchronization cycle timing. My naïve approach was to try to pass some sort of resolution map into the Reconcile function that would be used to pick the winner in a conflict, but there was the problem of passing this map to the synchronization loop at just the right time as well as managing the lifetime of the map (e.g. would resolutions continue to apply if disk errors prevented them from being carried out?).

Thinking about it now, an even "naïve-r" approach might be for a mutagen resume command to just pause the session and send a command to the losing endpoint telling it to delete the losing files. Then the normal conflict resolution would take over once the session resumed.

I'll let the idea stew for a bit longer, but it's on the radar. Let's see how the new synchronization modes shake out over this release.

[Toilal]

I wrote a python application mutagen-helper, just released today, that wraps mutagen binary and can help to manage your sessions.

Put configuration in a yaml file in a directory to sync, and run mutagen-helper up / mutagen-helper down to create / terminate sessions.

You may look at sources as it contains a class that can execute mutagen commands and parse output, but of course as soon as a public API will be available, it could be dropped :).

[xenoscopic]

One quick update on this front: Mutagen v0.9.0-beta2 brings label/selector support for sessions, which should make a number of scripting operations significantly easier (since there's no longer any need to track session IDs). It's designed to be comparible with the syntax that Kubernetes uses (since it actually re-uses the Kubernetes implementation).

There are also new -c/--configuration and --no-global-configuration flags documented in the release notes that might be useful for tools scripting Mutagen.

I'll leave this issue open as there is still more to be done here. In particular, I think it would be nice to be able to stream monitoring output in different formats.

[xenoscopic]

Mutagen v0.15 now supports custom output formatting (using Go templates) in its list and monitor commands, including support for JSON output. More information is available in the v0.15 release notes and is coming to the documentation site soon. I think that takes care of the bulk of this issue. I suspect there may be tweaks and additional feature requests on that front, so feel free to open other feature requests as necessary.