opam master can't run commands on ocaml < 4.08.0 and Xcode 12

[Blaisorblade]

On mac os x 10.15, opam master fails to run openssl, or sw_vers, while opam 2.0.7 works fine, and openssl and sw_vers are both available:

$ git describe
2.1.0-beta2-121-g51c160be
$ git branch
  2.0
  allow-git-fetch-pwd
  allow-git-fetch-pwd-backport
* master
$ ./configure && make lib-ext && make
$ dune exec --profile=release  --promote-install-files -- opam repo list
Fatal error:
"openssl": permission denied.

The error persists after make install.

The programs work.

$ which openssl
/usr/bin/openssl
$ ls -l $(which openssl )
-rwxr-xr-x 1 root wheel 510656 Sep 22 02:30 /usr/bin/openssl
$ openssl
OpenSSL> version
LibreSSL 2.8.3
OpenSSL> exit
$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.15.7
BuildVersion:	19H15

opam config report output, as requested:

$ dune exec --profile=release  --promote-install-files -- opam config report
# opam config report
# opam-version         2.1.0~beta4 (51c160be9a018b69ee2ba6cc38c483565bf00550)
# self-upgrade         no
Fatal error:
"sw_vers": permission denied.

[rjbou]

In both case, it shouldn't error with a simple permission denied.
For openssl, opam should fallback to internal sha computing. Could you share the stack trace of opam repo list ?
The code for permission check can help to locate the problem.

[Blaisorblade]

Could you share the stack trace of opam repo list ?

I'm afraid I have no idea how

The code for permission check can help to locate the problem.

Shouldn't that code use https://caml.inria.fr/pub/docs/manual-ocaml/libref/Unix.html#VALaccess instead? In C, access is implemented by a separate kernel API, which is not guaranteed to have the implementation you use.

[Blaisorblade]

Looking at 41d44f9 and ocaml/ocaml#7640, I see why you reimplement PATH lookup, but that shouldn't prevent you from using access(2).

[rjbou]

You should have the stack trace by setting OCAMLRUNPARAM environement variable : OCAMLRUNPARAM=b opam repo list

For acls, it is well documented in #4265 /cc @dra27

[dra27]

The ACL stuff shouldn’t be happening unless you’re on Cygwin - although it’s making me wonder if that’s strictly necessary there too, now (although I expect I would still have hit the same bug in Cygwin even using access!)

[Blaisorblade]

For openssl, opam should fallback to internal sha computing.

That part is fixed in #4449.

The permission errors remain — printf debugging suggests that getgroups throws an Invalid_argument("getgroups not implemented") — opam list attached below.

The ACL stuff shouldn’t be happening unless you’re on Cygwin - although it’s making me wonder if that’s strictly necessary there too, now (although I expect I would still have hit the same bug in Cygwin even using access!)

AFAIK Darwin has its own ACLs, but I wasn't asking about OpamACL.get_acl_executable_info (I have no idea about ACLs), just the rest of the access reimplementation. You've made it much better, but Unix systems have multiple ways to add extra permission checks, including SELinux.

Context:

$ id
uid=501(pgiarrusso) gid=20(staff) groups=20(staff),12(everyone),61(localaccounts),79(_appserverusr),80(admin),81(_appserveradm),98(_lpadmin),701(com.apple.sharepoint.group.1),33(_appstore),100(_lpoperator),204(_developer),250(_analyticsusers),395(com.apple.access_ftp),398(com.apple.access_screensharing),399(com.apple.access_ssh),400(com.apple.access_remote_ae)
$ opam list
# Packages matching: installed
# Name                   # Installed           # Synopsis
base                     v0.13.2               Full standard library replacement for OCaml
base-bigarray            base
base-bytes               base                  Bytes library distributed with the OCaml compiler
base-threads             base
base-unix                base
biniou                   1.2.1                 Binary data format designed for speed, safety, ease of use and backward compatibility as protocols evolve
cairo2                   0.6.1                 Binding to Cairo, a 2D Vector Graphics Library
cmdliner                 1.0.4                 Declarative definition of command line interfaces for OCaml
conf-adwaita-icon-theme  1                     Virtual package relying on adwaita-icon-theme
conf-cairo               1                     Virtual package relying on a Cairo system installation
conf-findutils           1                     Virtual package relying on findutils
conf-gtk3                18                    Virtual package relying on GTK+ 3
conf-gtksourceview3      0+2                   Virtual package relying on a GtkSourceView-3 system installation
conf-m4                  1                     Virtual package relying on m4
conf-perl                1                     Virtual package relying on perl
conf-pkg-config          1.3                   Virtual package relying on pkg-config installation
coq                      8.12.1+flambda-native pinned to version 8.12.1+flambda-native
coq-autosubst            dev.coq86             Autosubst is a Coq library for parallel de Bruijn substitutions
coq-ext-lib              0.11.2                A library of Coq definitions, theorems, and tactics
coq-iris                 3.3.0                 Iris is a Higher-Order Concurrent Separation Logic Framework with support for interactive proofs
coq-iris-string-ident    0.1.0                 Add support for Gallina names in intro patterns to the Iris Proof Mode
coq-lens                 1.0.1+8.12            Generation of lenses for record datatypes
coq-metacoq-template     1.0~beta1+8.12        A quoting and unquoting library for Coq in Coq
coq-serapi               8.12.0+0.12.0         Serialization library and protocol for machine interaction with the Coq proof assistant
coq-stdpp                1.4.0                 std++ is an extended "Standard Library" for Coq
coq-unicoq               1.5+8.12              An enhanced unification algorithm for Coq
coqdep                   8.12.0                custom coqdep package for Coq
coqide                   8.12.1                IDE of the Coq formal proof management system
cppo                     1.6.6                 Code preprocessor like cpp for OCaml
csexp                    1.3.2                 Parsing and printing of S-expressions in Canonical form
cudf                     0.9-1                 CUDF library (part of the Mancoosi tools)
dose3                    5.0.1                 Dose library (part of Mancoosi tools)
dune                     2.7.1                 Fast, portable, and opinionated build system
dune-configurator        2.7.1                 Helper library for gathering system configuration
easy-format              1.3.2                 High-level and functional interface to the Format module of the OCaml standard library
extlib                   1.7.7                 A complete yet small extension for OCaml standard library (reduced, recommended)
lablgtk3                 3.1.1                 OCaml interface to GTK+3
lablgtk3-sourceview3     3.1.1                 OCaml interface to GTK+ gtksourceview library
num                      1.4                   The legacy Num library for arbitrary-precision integer and rational arithmetic
ocaml                    4.07.1                The OCaml compiler (virtual package)
ocaml-compiler-libs      v0.12.3               OCaml compiler libraries repackaged
ocaml-config             1                     OCaml Switch Configuration
ocaml-migrate-parsetree  1.8.0                 Convert OCaml parsetrees between different versions
ocaml-secondary-compiler 4.08.1-1              OCaml 4.08.1 Secondary Switch Compiler
ocaml-variants           4.07.1+flambda        Official release 4.07.1, with flambda activated
ocamlbuild               0.14.0                OCamlbuild is a build system with builtin rules to easily build most OCaml projects.
ocamlfind                1.8.1                 A library manager for OCaml
ocamlfind-secondary      1.8.1                 ocamlfind support for ocaml-secondary-compiler
ocamlgraph               1.8.8                 A generic graph library for OCaml
opam-depext              1.1.5                 install OS distribution packages
opam-file-format         2.1.1                 Parser and printer for the opam file syntax
parsexp                  v0.13.0               S-expression parsing library
ppx_derivers             1.2.1                 Shared [@@deriving] plugin registry
ppx_deriving             5.1                   Type-driven code generation for OCaml
ppx_deriving_yojson      3.6.1                 JSON codec generator for OCaml
ppx_import               1.7.1                 A syntax extension for importing declarations from interface files
ppx_sexp_conv            v0.13.0               [@@deriving] plugin to generate S-expression conversion functions
ppx_tools_versioned      5.4.0                 A variant of ppx_tools based on ocaml-migrate-parsetree
ppxlib                   0.15.0                Standard library for ppx rewriters
re                       1.9.0                 RE is a regular expression library for OCaml
result                   1.5                   Compatibility Result module
seq                      base                  Compatibility package for OCaml's standard iterator type starting from 4.07.
sexplib                  v0.13.0               Library for serializing OCaml values to and from S-expressions
sexplib0                 v0.13.0               Library containing the definition of S-expressions and some base converters
stdlib-shims             0.1.0                 Backport some of the new stdlib features to older compiler
yojson                   1.7.0                 Yojson is an optimized parsing and printing library for the JSON format

[Blaisorblade]

This is interesting https://github.com/ocaml/ocaml/blob/28c337fecbebfc064c06ae0ba35a1107b0b58493/otherlibs/unix/getgroups.c#L20.

[Blaisorblade]

Okay, ocaml 4.07.1's configure gives (after instrumenting) getgroups() not found. The problem seems to boil down to 2 issues:

1. First, the configure test for getgroups is buggy (it should include unistd.h), and gcc got stricter about it:

$ cc -o tst getgroups.c
getgroups.c:24:7: error: implicit declaration of function 'getgroups' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
  if (getgroups(NGROUPS_MAX, gidset) == -1) return 1;
      ^
1 error generated.

2. The opam package uses -Wno-implicit-function-declaration, but it only patches "config/auto-aux/hasgot", while the offending test also uses ./runtest:

if sh ./hasgot -i limits.h && sh ./runtest getgroups.c; then

[Blaisorblade]

The new title is tentative and is based on extrapolation — I've estimated the impact based on ocaml/opam-repository@3d6ee07.

I can also confirm that adding -Wno-implicit-function-declaration to runtest restores a few missing functions — not just getgroups, but also setgroups, initgroups and "Asynchronous I/O".

Source patch (EDIT: now also Blaisorblade/ocaml@c4d1ba5).

diff --git a/config/auto-aux/runtest b/config/auto-aux/runtest
index c889a0dbd..861598e65 100755
--- a/config/auto-aux/runtest
+++ b/config/auto-aux/runtest
@@ -15,7 +15,7 @@
 #*                                                                        *
 #**************************************************************************

-cmd="$cc $cflags -o tst $* $ldflags $cclibs"
+cmd="$cc $cflags -Wno-implicit-function-declaration -o tst $* $ldflags $cclibs"

 if $verbose; then
   echo "runtest: $cmd" >&2

Effect on configure logs:

--- configure-before.log	2020-11-26 00:50:53.801818505 +0100
+++ configure-after.log	2020-11-26 00:51:14.434824601 +0100
@@ -41,7 +41,11 @@
 symlink() found.
 waitpid() found.
 wait4() found.
+getgroups() found.
+setgroups() found.
+initgroups() found.
 POSIX termios found.
+Asynchronous I/O are supported.
 setitimer() found.
 gethostname() found.
 uname() found.

[Blaisorblade]

FWIW: while the main issue is due to old OCaml versions, and the use of access is not the main question, I've checked two implementations of which, and they both seem based on access, but not as clearly as I expected; most likely any change would need additional testing. Posting the links I found for reference:

https://github.com/freebsd/freebsd/blob/c5a1b0313c98d82c7d73ace37d69cee697f25cbf/usr.bin/which/which.c#L105-L121 (there's a workaround for root, maybe because access pretends root has all permissions — but that seems more limited, and not clearly relevant for ocaml).
https://salsa.debian.org/debian/debianutils/-/blob/3e1637e1f54052d633cf99511f4b7049d864a259/which#L39 (uses test -x, which FreeBSD maps to (e)access: https://github.com/freebsd/freebsd/blob/c5a1b0313c98d82c7d73ace37d69cee697f25cbf/bin/test/test.c#L377-L389).

[bactrian]

dev meeting 26/2/21:

@Blaisorblade how important is support for OCaml <4.08 to you for compiling the opam binary? We are going to make 4.08 the minimum supported version for compiling opam 2.1.0 onwards.

Note that there is a make cold target that uses a bundled OCaml compiler if your system OCaml is too old, and that this does not affect older compilers for the purposes of opam switch

[avsm]

(woops, that was @avsm above in the wrong Firefox container ;-)

[Blaisorblade]

@bactrian OCaml 4.07.1+flambda is still one of the best compilers for Coq, but I need that in-switch. If I create another Opam patch, I’ll create a new switch.

[Blaisorblade]

And I suppose could also use ocaml-secondary-compiler like dune, tho I’d probably have to contribute that myself if I wanted.

[Blaisorblade]

oh I get it, meant to tag @avsm

[dra27]

@Blaisorblade - I'm not sure you do need a secondary compiler here. The issue is that the opam binary itself was built with OCaml 4.07, which you don't need to do, presumably?

[dra27]

Once you've built opam with OCaml 4.08+ you can use it to set-up whichever version of OCaml you like. It looks to me like this issue is resolved, therefore?

[Blaisorblade]

No, but nevermind.

[Blaisorblade]

Just for the record: ocaml-secondary-compiler would avoid the need to install another compiler for the only purpose of building opam development versions. That's the context for

If I create another Opam patch, I’ll create a new switch.

And I suppose could also use ocaml-secondary-compiler like dune, tho I’d probably have to contribute that myself if I wanted.