Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: document aqua/ubi better #3234

Merged
merged 1 commit into from
Nov 27, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 48 additions & 3 deletions docs/dev-tools/backends/aqua.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,25 @@
# Aqua Backend

[Aqua](https://aquaproj.github.io/) tools may be used natively in mise. Aqua is encouraged as a backend for new tools if they
cannot be used with ubi as aqua tools directly fetch tarballs from the vendor without requiring unsafe
code execution in a plugin.
[Aqua](https://aquaproj.github.io/) tools may be used natively in mise. aqua is the ideal backend
to use for new tools since they don't require plugins, they work on windows, they offer security
features like cosign/slsa verification in addition to checksums. aqua installs also show more progress
bars, which is nice.

You do not need to separately install aqua. The aqua CLI is not used in mise at all. What is used is
the [aqua registry](https://github.com/aquaproj/aqua-registry) which is a bunch of yaml files that get compiled into the mise binary on release.
Here's an example of one of these files: [`aqua:hashicorp/terraform`](https://github.com/aquaproj/aqua-registry/blob/main/pkgs/hashicorp/terraform/registry.yaml).
mise has a reimplementation of aqua that knows how to work with these files to install tools.

As of this writing, aqua is relatively new to mise and because a lot of tools are being converted from
asdf to aqua, there may be some configuration in aqua tools that need to be tightened up. I put some
common issues below and would strongly recommend contributing changes back to the aqua registry if you
notice problems. The maintainer is super responsive and great to work with.

If all else fails, you can disable aqua entirely with [`MISE_DISABLE_BACKENDS=aqua`](/configuration/settings.html#disable_backends).

Currently aqua plugins don't support setting environment variables or doing more than simply downloading
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jdx Rather than aqua plugins, do you perhaps mean "aqua tools" or "the aqua backend"?

In the first paragraph you mention that aqua doesn't require plugins:

aqua is the ideal backend to use for new tools since they don't require plugins

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes thank you

binaries though (and I'm not sure this functionality would ever get added), so some tools will likely
always require plugins like asdf/vfox.

The code for this is inside the mise repository at [`./src/backend/aqua.rs`](https://github.com/jdx/mise/blob/main/src/backend/aqua.rs).

Expand Down Expand Up @@ -32,3 +49,31 @@ to use the aqua backend. To see these tools, run `mise registry | grep aqua:`.
import Settings from '/components/settings.vue';
</script>
<Settings child="aqua" :level="3" />

## Common aqua issues

Here's some common issues I've seen when working with aqua tools.

### Supported env missing

The aqua registry defines supported envs for each tool of the os/arch. I've noticed some of these
are simply missing os/arch combos that are in fact supported—possibly because it was added after
the registry was created for that tool.

The fix is simple, just edit the `supported_envs` section of `registry.yaml` for the tool in question.

### Using `version_filter` instead of `version_prefix`

This is a weird one that doesn't causes weird issues in mise. In general in mise we like versions like
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering if this was meant to be "that causes" or "that doesn't cause", now it's a bit between the two :)

`1.2.3` with no decoration like `v1.2.3` or `cli-v1.2.3`. This consistency not only makes `mise.toml`
cleaner but, it also helps make things like `mise up` function right because it's able to parse it as
semver without dealing with a bunch of edge-cases.

Really if you notice aqua tools are giving you versions that aren't simple triplets, it's worth fixing.

One common thing I've seen is registries using a `version_filter` expression like `Version startsWith "Version startsWith "atlascli/""`.

This ultimately causes the version to be `atlascli/1.2.3` which is not what we want. The fix is to use
`version_prefix` instead of `version_filter` and just put the prefix in the `version_prefix` field.
In this example, it would be `atlascli/`. mise will automatically strip this out and add it back in,
which it can't do with `version_filter`.
27 changes: 7 additions & 20 deletions docs/dev-tools/backends/asdf.md
Original file line number Diff line number Diff line change
@@ -1,25 +1,12 @@
# asdf Backend

asdf is the original backend for mise. It's the default if no backend is specified,
e.g.: `mise i ripgrep` will use [asdf](https://gitlab.com/wt0f/asdf-ripgrep) but `mise i cargo:ripgrep`
will use the [cargo](./cargo) backend. You can explicitly specify the asdf backend with `mise i asdf:ripgrep`.
If you wish.

If choosing a backend to integrate a tool into mise, it's discouraged to use the asdf backend. ubi
would be the ideal choice if it can work as a single binary, otherwise aqua would be the next best choice
since it requires minimal configuration and doesn't require executing code in a plugin. Generally
vfox plugins can handle anything an asdf plugin might need to do while also being potentially able
to support windows.

There are [hundreds of plugins](https://github.com/mise-plugins/registry) available in the
[mise registry](https://github.com/mise-plugins) and you can also install plugins from git
repos or local directories.

::: warning
Take care using plugins as well as anything else you get from the internet. CLIs are
unfortunately capable of doing a lot of damage to your system if a bad actor manages to
get into your system through a plugin or other tool.
:::
asdf is the original backend for mise. It relies on asdf plugins for each tool. asdf plugins are
more risky to use because they're typically written by a single developer unrelated to the tool vendor
they also do not function on Windows.
Because of the extra complexity of asdf tools and security concerns we are actively moving tools in
the registry away from asdf where possible to backends like aqua and ubi which don't require plugins.
That said, not all tools can function with ubi/aqua if they have a unique installation process or
need to set env vars other than PATH.

## Writing asdf plugins for mise

Expand Down
61 changes: 59 additions & 2 deletions docs/dev-tools/backends/ubi.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,12 @@
# Ubi Backend

You may install GitHub Releases and URL packages directly using [ubi](https://github.com/houseabsolute/ubi) backend. ubi is directly compiled into
the mise codebase so it does not need to be installed separately to be used. ubi is the preferred backend when it functions for tools as it is the
simplest and requires minimal configuration.
the mise codebase so it does not need to be installed separately to be used. ubi is preferred over
asdf/vfox for new tools since it doesn't require a plugin, supports Windows, and is really easy to use.

ubi doesn't require plugins or even any configuration for each tool. What it does is try to deduce what
the proper binary/tarball is from GitHub releases and downloads the right one. As long as the vendor
uses a somewhat standard labeling scheme for their releases, ubi should be able to figure it out.

The code for this is inside of the mise repository at [`./src/backend/ubi.rs`](https://github.com/jdx/mise/blob/main/src/backend/ubi.rs).

Expand Down Expand Up @@ -59,3 +63,56 @@ then this will be ignored.
- **GitHub shorthand for latest release version:** `ubi:goreleaser/goreleaser`
- **GitHub shorthand for specific release version:** `ubi:goreleaser/goreleaser@1.25.1`
- **URL syntax:** `ubi:https://github.com/goreleaser/goreleaser/releases/download/v1.16.2/goreleaser_Darwin_arm64.tar.gz`

## Troubleshooting ubi

### `ubi` resolver can't find os/arch

Sometimes vendors use strange formats for their releases that ubi can't figure out, possibly for a
specific os/arch combination. For example this recently happend in [this ticket](https://github.com/houseabsolute/ubi/issues/79) because a vendor used
"mac" instead of the more common "macos" or "darwin" tags.

Try using ubi by itself to see if the issue is related to mise or ubi:

```sh
ubi -p jdx/mise
./bin/mise -v # yes this technically means you could do `mise use ubi:jdx/mise` though I don't know why you would
```

### `ubi` picks the wrong tarball

Another issue is that a GitHub release may have a bunch of tarballs, some that don't contain the CLI
you want, you can use the `matching` field in order to specify a string to match against the release.

```sh
mise use ubi:tamasfe/taplo[matching=full]
# or with ubi directly
ubi -p tamasfe/taplo -m full
```

### `ubi` can't find the binary in the tarball

ubi assumes that the repo name is the same as the binary name, however that is often not the case.
For example, BurntSushi/ripgrep gives us a binary named `rg` not `ripgrep`. In this case, you can
specify the binary name with the `exe` field:

```sh
mise use ubi:BurntSushi/ripgrep[exe=rg]
# or with ubi directly
ubi -p BurntSushi/ripgrep -e rg
```

### `ubi` uses weird versions

This issue is actually with mise and not with ubi. mise needs to be able to list the available versions
of the tools so that "latest" points to whatever is the actual latest release of the CLI. What sometimes
happens is vendors will have GitHub releases for unrelated things. For example, `cargo-bins/cargo-binstall`
is the repo for cargo-binstall, however it has a bunch of releases for unrelated CLIs that are not
cargo-binstall. We need to filter these out and that can be specified with the `tag_regex` tool option:

```sh
mise use 'ubi:cargo-bins/cargo-binstall[tag_regex=^\d+\.]'
```

Now when running `mise ls-remote ubi:cargo-bins/cargo-binstall[tag_regex=^\d+\.]` you should only see
versions starting with a number. Note that this command is cached so you likely will need to run `mise cache clear` first.
16 changes: 16 additions & 0 deletions docs/registry.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,22 @@ editLink: false

# Registry

In general, the preferred backend to use for new tools is the following:

- [aqua](./dev-tools/backends/aqua.html) - offers the most features and security while not requiring plugins
- [ubi](./dev-tools/backends/ubi.html) - very simple to use
- [pipx](./dev-tools/backends/pipx.html) - only for python tools, requires python to be installed but this generally would always be the case for python tools
- [npm](./dev-tools/backends/npm.html) - only for node tools, requires node to be installed but this generally would always be the case for node tools
- [vfox](./dev-tools/backends/vfox.html) - only for tools that have unique installation requirements or need to modify env vars
- [asdf](./dev-tools/backends/asdf.html) - only for tools that have unique installation requirements or need to modify env vars, doesn't support windows
- [go](./dev-tools/backends/go.html) - only for go tools, requires go to be installed to compile. Because go tools can be distributed as a single binary, aqua/ubi are definitely preferred.
- [cargo](./dev-tools/backends/cargo.html) - only for rust tools, requires rust to be installed to compile. Because rust tools can be distributed as a single binary, aqua/ubi are definitely preferred.

However, each tool can define its own priority if it has more than 1 backend it supports. You can disable a backend with `mise settings disable_backends=asdf`.
And it will be skipped. See [Aliases](/dev-tools/aliases.html) for a way to set a default backend for a tool.

You can also specify the full name for a tool using `mise use aqua:1password/cli` if you want to use a specific backend.

| Short | Full |
| ----------- | --------------- |
| 1password | [asdf:NeoHsu/asdf-1password-cli](https://github.com/NeoHsu/asdf-1password-cli) [aqua:1password/cli](https://github.com/1password/cli) |
Expand Down
16 changes: 16 additions & 0 deletions scripts/render-registry.js
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,22 @@ editLink: false
---

# Registry

In general, the preferred backend to use for new tools is the following:

- [aqua](./dev-tools/backends/aqua.html) - offers the most features and security while not requiring plugins
- [ubi](./dev-tools/backends/ubi.html) - very simple to use
- [pipx](./dev-tools/backends/pipx.html) - only for python tools, requires python to be installed but this generally would always be the case for python tools
- [npm](./dev-tools/backends/npm.html) - only for node tools, requires node to be installed but this generally would always be the case for node tools
- [vfox](./dev-tools/backends/vfox.html) - only for tools that have unique installation requirements or need to modify env vars
- [asdf](./dev-tools/backends/asdf.html) - only for tools that have unique installation requirements or need to modify env vars, doesn't support windows
- [go](./dev-tools/backends/go.html) - only for go tools, requires go to be installed to compile. Because go tools can be distributed as a single binary, aqua/ubi are definitely preferred.
- [cargo](./dev-tools/backends/cargo.html) - only for rust tools, requires rust to be installed to compile. Because rust tools can be distributed as a single binary, aqua/ubi are definitely preferred.

However, each tool can define its own priority if it has more than 1 backend it supports. You can disable a backend with \`mise settings disable_backends=asdf\`.
And it will be skipped. See [Aliases](/dev-tools/aliases.html) for a way to set a default backend for a tool.

You can also specify the full name for a tool using \`mise use aqua:1password/cli\` if you want to use a specific backend.
`,
];

Expand Down