depexts integration #3975

rjbou · 2019-09-02T14:26:06Z

Integration of external dependencies management in opam: System dependencies are retrieved, checked and user is warned before build failure.
What is added ?

① At switch load, we retrieve the depext state of available packages. They are marked as:

installed
available: not installed but available on current repositories. When the availability can't be retrieved, all not installed packages are marked as available (like on gentoo or *bsd)
not_found: not found on available repositories, e.g. on debian only current registered repositories are checked (source.list*).
For more information on what commands are used, see this function.

② It is then used to check that installed opam packages still have their system dependencies installed. If no, their are removed from installed packages and marked for reinstall (like when an external file dependencies changes).

③ It is also used at install, before performing actions, depexts of packages to install are checked. If some are missing, there is several interactions: proposal to install them via opam, proposal of the command line to run to install them, nothing, etc. All these interactions can be instrumented via the cli, using options depext-no-root, --depext-print-only

④ It is possible to notify opam to bypass the check of some system packages, using the option --depext-bypass. The information is stored in the switch config for further operations.

src/client/opamArg.ml

src/state/opamStateConfig.mli

src/state/opamStateTypes.mli

src/state/opamSwitchState.ml

src/state/opamSysInteract.ml

rjbou · 2019-09-11T13:08:56Z

src/state/opamSysInteract.ml

+(*  exception on linking described in the file LICENSE.                   *)
+(*                                                                        *)
+(**************************************************************************)
+


Change module name?

rjbou · 2019-10-10T12:51:58Z

close #3790

src/state/opamSysInteract.ml

rjbou · 2019-10-29T17:27:41Z

check dry run
depends on config: add 'config setopt' and 'setopt' (sub)commands to modify global config fields #3992 for configuration
depends on Define switch invariants rather than "base packages" #3894 for sys_packages lazyness
clean branch
depends on Option configuration, v2 #4116 for configuration

rjbou · 2019-12-18T22:21:46Z

related #3692
close #1519
close #2426

hannesm · 2020-03-22T11:00:50Z

I have not looked into this PR in detail, but what I currently am missing from depext is a way to figure out which versions of the external dependencies are installed. As example: conf-pkg-config depends on the pkg-config debian package. Now, for reproducible builds I'd like to capture "pkg-config version 0.29-4" is installed (to emit this information into a build-info, and verify on rebuilding time that the package versions did not change). What would be great would be the output of a complete "package name, version" list that includes all transitive dependencies. NB: this does by no means should stop this PR, as the additional functionality can be provided separately; I just wanted to describe what from my perspective is desired for reproducibility :)

AltGr · 2020-03-23T08:43:13Z

@hannesm: indeed, we don't have that kind of information at the moment.
Your best bet would be to use opam list -e, then call your system package manager manually on them to query their installed versions or checksums.

hannesm · 2020-03-23T08:48:01Z

@AltGr ok, If I understand correctly, depext is to abstract away the host system package managers, and once this PR is in, I'd be keen to extend depext with an "info" subcommand (which gets a list of packages and returns a list of packages + versions).

plus some fixes and spelling.

* adds depext_bypass to switch_config * when a package is installed successfully after bypassing its depexts, save the depext-bypass so that the user won't be asked again every time NOTE: the new field is documented, but refers to an undocumented global config field!

that allows to register a given sysdep as an exception, iff the corresponding package installs successfully. This might be slightly inaccurate if several packages use the same sys dependency and one has a broken check, but I don't think that's something to worry about.

So they don't get pissed about being asked each time ;)

... and implement the `depext-no-root` option The combination with `--skip-depexts` is not yet implemented

! `~/.opam/config` / `gt.config` only - `depext: true` Enable handling of external dependencies. When this is enabled, opam will interact with your system package manager when packages declare external dependencies, and prompt you to install them when needed. - `depext-run-installs: true` (was: `print-only`) Allows opam to run package installations through your system package manager when required by "depexts" of opam packages. You will be prompted before any calls to `sudo`, unless `--yes` was specified; if `false`, the installation command will be printed, and opam will pause to let you proceed. - `depext-cannot-install: false` (was: `no-root`) Instructs opam that no system package can be installed on your system. Any opam package declaring system dependencies towards a system package that is not yet installed will be marked as unavailable. Use this if you don't have root access. - `depext-verify: true` (was: `no-consistency-checks`) If an installed opam package depends on a system package that was removed and this is enabled, opam will mark the opam package as needing reinstallation. `~/.opam/config` + switch-config: - `depext-bypass: [SYS_PKGS]` List of system packages to assume as installed, without considering the results from the system package manager. ! CLI - `--assume-depexts` (OpamClientConfig) Skip the installation step for any missing system packages, and attempt to proceed with compilation of the opam packages anyway. If the installation is successful, opam won't prompt again about these system packages. - `--no-depexts` (OpamStateConfig) Temporarily disables handling of external dependencies. Implies `--assume-depexts`.

which doesn't make much sense in our latest iteration, and is to complicated to explain. Let's wait to see exactly what need arises if something like it is to be done at some point.

AltGr reviewed Sep 3, 2019

View reviewed changes

src/client/opamArg.ml Outdated Show resolved Hide resolved

AltGr reviewed Sep 4, 2019

View reviewed changes

rjbou added the PR: WIP Not for merge at this stage label Sep 5, 2019

rjbou commented Sep 11, 2019

View reviewed changes

rjbou force-pushed the depext branch from 7dad041 to e64c9ec Compare September 12, 2019 09:54

rjbou force-pushed the depext branch from e64c9ec to 774e38e Compare September 25, 2019 14:53

rjbou commented Oct 29, 2019

View reviewed changes

src/state/opamSysInteract.ml Outdated Show resolved Hide resolved

rjbou added the PR: QUEUED Pending pull request, waiting for other work to be merged or closed label Oct 29, 2019

rjbou force-pushed the depext branch 3 times, most recently from c63324b to c613a38 Compare November 25, 2019 17:31

carlosdagos mentioned this pull request Dec 1, 2019

Make 2.0 the default branch ocaml-opam/opam-depext#122

Closed

kit-ty-kate mentioned this pull request Dec 4, 2019

[new release] lablgtk3-gtkspell3, lablgtk3-sourceview3 and lablgtk3 (3.0.beta8) ocaml/opam-repository#15439

Merged

rjbou mentioned this pull request Dec 9, 2019

opam depext integration #3790

Closed

rjbou force-pushed the depext branch from 2905f47 to 9957238 Compare December 18, 2019 16:36

rjbou added this to the 2.1.0 milestone Jan 7, 2020

rjbou removed the PR: WIP Not for merge at this stage label Jan 24, 2020

rjbou requested a review from AltGr January 27, 2020 13:59

rjbou force-pushed the depext branch 2 times, most recently from ce997a6 to 79598d4 Compare March 12, 2020 11:25

rjbou force-pushed the depext branch 3 times, most recently from 62e4b24 to ae11de9 Compare March 30, 2020 15:12

rjbou force-pushed the depext branch from 7d21064 to ee8f901 Compare April 10, 2020 12:28

rjbou and others added 23 commits April 14, 2020 21:01

syspoll: add a to_string for config printing

d97ce53

depext: refactor install management

8b6fb19

sysinteract: don't confirm before sudo

6cda94a

sysinteract: change return type of packages_status

7952d75

sysinteract: not used installed

e036aa4

sysinteract: handle ':' for debian

f58c5a1

depext: update tests suite with real packages

ed2c70b

switchstate: lazy sys_packages

ae51f03

depext configuration: add depxt options to setopt modifiable fields

7eaadf3

depext: handle correctly dry run, fake & show

9b831f7

depext: allow retry when install fail

2bc99e7

sysinteract: don't propagate --yes to install system command

44a6da6

depext: dry run & fake handling on package status retrieving

75e250e

Rework the 'depext' user interactions

a43530c

plus some fixes and spelling.

Disable depext hint when auto-depexts enabled

f5bcf15

Fix print_as_table bug

edb38e6

Print hint when the user refuses to let opam run sys commands

987f010

So they don't get pissed about being asked each time ;)

[needs more testing] use sys-deps availability for package availability

dc3b2c3

... and implement the `depext-no-root` option The combination with `--skip-depexts` is not yet implemented

Remove the 'depext-verify' option

7ef2027

which doesn't make much sense in our latest iteration, and is to complicated to explain. Let's wait to see exactly what need arises if something like it is to be done at some point.

depext,doc: add note in FAQ

12839dc

rjbou force-pushed the depext branch from d71d2e6 to 12839dc Compare April 14, 2020 19:08

AltGr merged commit fe493cf into ocaml:master Apr 15, 2020

rjbou linked an issue May 22, 2020 that may be closed by this pull request

opam depext integration #3790

Closed

rjbou mentioned this pull request Jun 24, 2020

Automatically check depexts field #1519

Closed

dra27 mentioned this pull request Jul 13, 2020

Restore `Denied behaviour of PATH resolution #4265

Merged

kit-ty-kate mentioned this pull request Sep 13, 2020

[opam 2.1~beta2] opam 2.1 does not seem to have an equivalant for opam depext -u #4355

Closed

hannesm mentioned this pull request Jun 29, 2021

avoid capturing the Unix.environment at top-level #4731

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

depexts integration #3975

depexts integration #3975

rjbou commented Sep 2, 2019 •

edited

Loading

rjbou Sep 11, 2019

rjbou commented Oct 10, 2019 •

edited

Loading

rjbou commented Oct 29, 2019 •

edited

Loading

rjbou commented Dec 18, 2019 •

edited

Loading

hannesm commented Mar 22, 2020

AltGr commented Mar 23, 2020

hannesm commented Mar 23, 2020

depexts integration #3975

depexts integration #3975

Conversation

rjbou commented Sep 2, 2019 • edited Loading

rjbou Sep 11, 2019

Choose a reason for hiding this comment

rjbou commented Oct 10, 2019 • edited Loading

rjbou commented Oct 29, 2019 • edited Loading

rjbou commented Dec 18, 2019 • edited Loading

hannesm commented Mar 22, 2020

AltGr commented Mar 23, 2020

hannesm commented Mar 23, 2020

rjbou commented Sep 2, 2019 •

edited

Loading

rjbou commented Oct 10, 2019 •

edited

Loading

rjbou commented Oct 29, 2019 •

edited

Loading

rjbou commented Dec 18, 2019 •

edited

Loading