Prepare menuinst for multiplatform support

cep-9999.md

@@ -0,0 +1,348 @@
+<table>
+<tr><td> Title </td><td> Prepare<code>menuinst</code> for multiplatform support</td>
+<tr><td> Status </td><td> Draft</td></tr>
+<tr><td> Author(s) </td><td> Jaime Rodríguez-Guerra &lt;jaime.rogue@gmail.com&gt;</td></tr>
+<tr><td> Created </td><td> Oct 14, 2021</td></tr>
+<tr><td> Updated </td><td> Oct 18, 2021</td></tr>
+<tr><td> Discussion </td><td> NA </td></tr>
+<tr><td> Implementation </td><td> NA </td></tr>
+</table>
+
+## Abstract
+
+`menuinst` is a library used by `conda` to install menu items that point to commands provided by
+`conda` packages. It operates by discovering certain JSON files located in `$PREFIX/Menu` after
+linking the package to the environment.
+
+This library has supported Windows for a long time but support for Linux and MacOS was not there
+(even if the original project did offer support). As these platforms are brought up to speed, we
+have an opportunity to fix some existing issues with the project and its integations. Namely:
+
+1. Providing the same Python API for all platforms
+2. Consolidating the `conda-standalone` / `constructor` CLI integrations in `conda` itself
+3. Consolidating the JSON metadata required in each OS and formalize it in a schema so other
+   implementations can validate it with automated tooling.
+
+
+## Python API
+
+### Current state
+
+`menuinst` provides a Python API comprised of a single function `menuinst.install(...)`
+with the following signature:
+
+```python
+def install(path, remove=False, prefix=sys.prefix, recursing=False, root_prefix=sys.prefix):
+  ...
+```
+
+> Note that calling `menuinst.install` for platforms other than Windows currently fails due to the
+extra `root_prefix` argument added for `conda-standalone` compatibility.
+
+Other projects provide a layer on top of `menuinst.install`:
+
+* `constructor/nsis/_nsis.py` [introduces](https://github.com/conda/constructor/blob/master/constructor/nsis/_nsis.py#L107)
+  `mk_menus` and `rm_menus`, which iterate `PREFIX/Menu` looking for suitable JSON files.
+
+```python
+def mk_menus(remove=False, prefix=None, pkg_names=None, root_prefix=None):
+    ...
+
+def rm_menus(prefix=None, root_prefix=None):
+    ...
+```
+
+* `conda` [provides](https://github.com/conda/conda/blob/33a142c16530fcdada6c377486f1c1a385738a96/conda/gateways/disk/create.py#L231-L248)
+ `make_menu`, which actively excludes Linux / MacOS.
+
+```python
+def make_menu(prefix, file_path, remove=False):
+    ...
+```
+
+### Motivation
+
+The official API is confusing (install can also remove) and leaves opportunity for downstream to
+create their own logic for basic functionality. `menuinst` should be the sole provider here.
+
+
+### Suggested actions
+
+`menuinst` needs to provide two functions: `install` and `remove`.
+
+```python
+def install(metadata: [str, dict], target_prefix: str = sys.prefix, root_prefix: str = sys.prefix) -> List[str]:
+  ...
+
+def remove(metadata: [str, dict], target_prefix: str = sys.prefix, root_prefix: str = sys.prefix) -> List[str]:
+  ...
+```
+
+`metadata` can be either a path to a JSON file, a file handler or an already parsed object.
+Regardless the source, the resulting metadata object needs to satisfy the metadata schema.
+Both functions return a list of strings that point to the path of the menu items created or removed.
+
+Additionally, an utility function for prefixes could provide:
+
+```python
+def install_all(prefix: str, root_prefix: str = sys.prefix, filter: callable = None) -> List[List[str]]:
+  ...
+
+def remove_all(prefix: str, root_prefix: str = sys.prefix, filter: callable = None) -> List[List[str]]:
+  ...
+```
+
+It should be discussed whether:
+
+* `root_prefix` is only used internally by `conda-standalone` / `constructor` and hence should be
+  made a private name `_root_prefix`.
+* `recursing` is needed on Windows or not.
+* A list (of lists) of strings is the best way to return the created shortcuts. Some platforms
+  create a directory with shortcuts (Windows), some create several directories (MacOS), some create
+  a list of files in a given location (Linux).
+
+### Backwards compatibility
+
+New `conda` releases will pin accordingly. Releasing this API would entail a new major version, which
+should be enough to prevent breakages. Current version `4.10.3` is [pinning](https://github.com/conda-forge/conda-feedstock/blob/master/recipe/meta.yaml#L46)
+`menuinst` as `>=1.4.11,<2`.
+
+
+## CLI API
+
+### Current state
+
+`menuinst` does provide a CLI through [`menuinst.main`](https://github.com/conda/menuinst/blob/9ae6c95/menuinst/main.py):
+
+```console
+$> menuinst [--remove] [--prefix=sys.prefix] [json_file json_file ...]
+```
+
+However, `constructor` needs a way to call it from `conda-standalone`, which prompted this project
+to provide a [custom entry point](https://github.com/conda-forge/conda-standalone-feedstock/blob/834590a/recipe/entry_point.py#L25-L88)
+that mimics a `conda` sub-command:
+
+```console
+$> conda-standalone constructor [--prefix] [--make-menus|--rm-menus] ...
+```
+> This is needed because, even with `menuinst` own CLI, `conda-standalone` is a `pyinstaller`-frozen
+> executable which cannot call arbitrary entry points in the bundle.
+
+### Motivation
+
+A standard `menuinst` CLI (standalone or through conda) is needed and must not be provided as part
+of the `conda-standalone` project only. It is a surprising behaviour that needs to be replicated
+by other implementations and, as such, should be part of the main codebase.
+
+> `constructor` allows a different standalone conda executable to be bundled through the `--conda-exe`
+option, which opens the possibility of using `micromamba` as the bundled conda. The `constructor`
+sub-command is expected though, so `micromamba` needs to provide it as well.
+
+### Suggested actions
+
+Provide a `conda constructor` subcommand, officially part of the `conda` core. It should implement
+the same CLI as currently found in the `constructor` entry point, delegating the menu creation
+and removal to the `menuinst` public API. This CLI should be standardized so other projects can
+implement it if needed.
+
+`menuinst` itself does not need to provide a CLI, but if it does, it should be as similar as possible
+to the `constructor` counterparts.
+
+
+## Menu metadata schema
+
+### Current state
+
+The required metadata for each platform is documented in [the `menuinst` wiki](https://github.com/conda/menuinst/wiki/Menu-Shortcut-Config-Structure/632fbc84251c8a8093e1b56b0b5314d23c1e946b).
+However, only Windows is really supported by the tool. This assymmetrical growth has allowed Windows
+to grow an ad-hoc specification that doesn't really translate well to other platforms.
+
+The overall schema seems to be:
+
+```python
+{
+  "menu_name": str,
+  "menu_items": list of dict,
+}
+```
+
+Unfortunately, each menu item dict (let's call it `MenuItem`) takes a different form in each platform.
+
+#### `MenuItem` on Windows
+
+```python
+{
+  ["system" | "script" | "pyscript" | "pywscript" | "webbrowser"]: str,
+  "scriptargument": str,
+  "scriptarguments": list of str,
+  "name": str,
+  "workdir": str,
+  "icon": str,
+  "desktop": bool,
+  "quicklaunch": bool,
+}
+```
+
+Currently allowed placeholders are:
+
+* `${PREFIX}`: Python environment prefix
+* `${ROOT_PREFIX}`: Python environment prefix of root (conda or otherwise) installation
+* `${PYTHON_SCRIPTS}`: Scripts folder in Python environment, `${PREFIX}/Scripts`
+* `${MENU_DIR}`: Folder for menu config and icon files, `${PREFIX}/Menu`
+* `${PERSONALDIR}`: Not sure
+* `${USERPROFILE}`: User's home folder
+* `${ENV_NAME}`: The environment in which this shortcut lives.
+* `${DISTRIBUTION_NAME}`: The name of the folder of the root prefix, for example "Miniconda" if
+  distribution installed at "C:\Users\Miniconda".
+* `${PY_VER}`: The Python major version only. This is taken from the root prefix. Used generally
+  for placing shortcuts under a menu of a parent installation.
+* `${PLATFORM}`: one of (32-bit) or (64-bit). This is taken from the root prefix. Used generally
+  for placing shortcuts under a menu of a parent installation.
+
+#### `MenuItem` on MacOS
+
+```python
+{
+    "cmd": str
+    "name": str,
+    "icns": str,
+}
+```
+
+Currently allowed placeholders are:
+
+* `${BIN_DIR}`: `PREFIX/bin`
+* `${MENU_DIR}`: `PREFIX/Menu`
+
+#### `MenuItem` on Linux
+
+```python
+{
+    "cmd": list of str,
+    "id": str,
+    "name": str,
+    "comment": str.
+    "terminal": bool,
+    "icon": str,
+},
+```
+
+On Linux, only `cmd` can take two special placeholders `{{FILEBROWSER}}` and `{{WEBBROWSER}}`, which are
+replaced by the default Desktop file explorer, and the default web browser, respectively.
+
+
+### Motivation
+
+#### Standardize the `command` interface
+
+Windows has several ways to specify which command should be run with the shortcut:
+
+* `system` + `scriptargument[s]`: path to executable plus its arguments.
+* `script` + `scriptargument[s]`: same as above, but the executable is run in a subprocess after
+  invoking `ROOT_PYTHON cwp.py PREFIX`.
+* `pyscript`: hardcodes `script` to be `PREFIX/python.exe` and takes the value as the first (and
+  only) argument.
+* `pywscript`: same as above, but uses `pythonw.exe` as the launcher to, theoretically, avoid
+  launching a console window next to your application.
+* `webbrowser`: alias to `PREFIX/python -m webbrowser -t URL`.
+
+On Linux the command is specified with `cmd`, expressed as a list of strings. On MacOS, `cmd` is
+also taken, but in this case it's expected to be a raw string.
+
+#### Standardize the `icon` key
+
+Windows and Linux expect `icon`. MacOS expects `icns`. Each platform requires a different file format,
+but that can be arranged with placeholders.
+
+#### Standardize the placeholders
+
+Allowed placeholders vary vastly across platforms. A common subset must be identified and implemented.
+Per-platform options are allowed but only when strictly necessary.
+
+### Suggested actions
+
+#### New validated schema
+
+The current (unstandardized) schema will be considered version 1.0. The new schema will be versioned
+as 2.0, as identified by `$schema`:
+
+```python
+{
+  "$schema": str,
+  "menu_name": str,
+  "menu_items": [
+    {
+      "name": str,
+      "description": str,
+      "icon": str,
+      "command": list of str,
+      "working_directory": str,
+      "platform_options": {
+        "windows": {
+          "desktop": bool,
+          "quicklaunch": bool,
+        },
+        "linux": {
+          "terminal": bool,
+          "id": str,  # we might compute this on the fly
+        },
+        "osx": {},
+      }
+    }
+  ]
+}
+```
+
+Each JSON file will be validated against its `$schema`.
+
+#### Allowed placeholders
+
+Each platform must provide these placeholders, to be used in the value of any `str`-accepting key.
+Note that a Jinja-like syntax is used now instead of the legacy shell-like syntax. This is done to
+prevent assumptions on non-Unix platforms.
+
+* `{{ BASE_PREFIX }}`: Path to the base (conda or otherwise) installation.
+* `{{ DISTRIBUTION_NAME }}`: The name of the folder of the base prefix, for example "Miniconda" if
+   distribution installed at `C:\Users\Miniconda` or `/usr/local/miniconda`.
+* `{{ BASE_PYTHON }}`: Path to the Python interpreter found in the base installation.
+* `{{ PREFIX }}`: Path to the environment where the shortcut metadata was found.
+* `{{ ENV_NAME }}`: The environment name in which this shortcut comes from.
+* `{{ PYTHON }}`: Path to the Python interpreter installed in PREFIX.
+* `{{ MENU_DIR }}`: Folder for menu config and icon files, `{{ PREFIX }}/Menu`.
+* `{{ BIN_DIR }}`: Folder for menu config and icon files, `{{ PREFIX }}/bin` on Unix;
+  `{{ PREFIX }}\Library\bin` on Windows.
+* `{{ SP_DIR }}`: `{{ PREFIX }}/lib/pythonX.Y/site-packages` on Unix. `{{ PREFIX }}\Lib\site-packages` on
+  Windows.
+* `{{ PY_VER }}`: The Python major version only. This is taken from the root prefix.
+* `{{ HOME }}`: Personal directory of the user.
+* `{{ ICON_EXT }}`: `ico` on Windows, `png` on Linux, `icns` on MacOS.
+
+Windows only:
+
+* `{{ SCRIPTS_DIR }}`: Scripts folder in Python environment, `{{ PREFIX }}/Scripts`.
+* `{{ CWP }}`: Path to the `cwp.py` script used for environment pseudo-initialization in shortcuts.
+* `{{ PYTHONW }}`: Path to `pythonw.exe` in PREFIX.
+* `{{ BASE_PYTHONW }}`: Path to `pythonw.exe` in BASE_PREFIX.
+
+> Note: `cwp.py` will be augmented with command-line arguments to implement additional behaviour
+> that was only available beforehand through custom logic in the `script`-like options.
+
+
+### Backwards compatibility
+
+Windows users do depend on the existing schema. There are a lot of packages that use it. This
+(unversioned) document needs to be kept around and respected. In the absence of the `$schema` key,
+it will be assumed that the metadata is built with the legacy schema. This will be validated too,
+but will only throw warnings to prevent badly formed documents from being used successfully.
+
+## References
+
+* [Rework linux/osx support plus new simplified format?](https://github.com/conda/menuinst/issues/48)
+* [Mamba's implementation in C++](https://github.com/mamba-org/mamba/blob/81a490a/src/core/menuinst.cpp)
+* [Interactions between conda, conda-standalone, constructor and menuinst](https://gist.github.com/jaimergp/7de5843421d63fa4a408ac5c8712c3c9)
+* [Change the API to menuinst.install(path_or_dict)](https://github.com/conda/menuinst/issues/25)
+* [`menuinst` wiki as of 2021.10.18](https://github.com/conda/menuinst/wiki/Menu-Shortcut-Config-Structure/632fbc84251c8a8093e1b56b0b5314d23c1e946b)
+
+## Copyright
+
+All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/).