[opam 2.1] opam list --installed is really slow

[cmovcc]

With opam 2.0.9, consistently below 100ms :

$ time opam list --installed
# Packages matching: installed
[...]
real	0m0.066s
user	0m0.052s
sys	0m0.014s

With opam 2.1.0, around 10s on first query and then around 500ms:

$ time opam list --installed
# Packages matching: installed
[...]
real	0m9.259s
user	0m7.558s
sys	0m1.893s
$ time opam list --installed
# Packages matching: installed
[...]
real	0m0.568s
user	0m0.483s
sys	0m0.083s

The same behaviour can be observed with opam 2.1.1 (and when building current master).

$ opam --version
2.1.1
$ time opam list --installed
# Packages matching: installed
[...]
real	0m9.279s
user	0m7.767s
sys	0m1.681s

I had some trouble bisecting the issue, due to incompatible dune versions. I can take additional time for this if needed.
I stumbled onto this issue due to what is inserted by opam-user-setup in ~/.vimrc, leading to vim starting quite slowly.
Thank you very much for opam!

Addendum: While trying to reproduce this issue, I have found #4311. Thus, I search for https://github.com/ocaml-opam/opam-rt and was wondering: would it be reasonable to add performance regression tests for such issues? With respect to the CI machine potentially changing, etc, it seemed non-trivial to me to have reliable CI about this. I am interested in possible hints about this, and can create another issue on the opam-rt repo if this is off-topic here. Thank you again!

[rjbou]

What is you distro? Can you share the opam config report output ?

[rjbou]

for vim, you can find in this issue ocaml-opam/opam-user-setup#53 other way to speedup loading.

[cmovcc]

Sorry, I completely missed #4875 .
I use Arch Linux.

$ opam config report
# opam config report
# opam-version         2.1.1 
# self-upgrade         no
# system               arch=x86_64 os=linux os-distribution=arch os-version=rolling
# solver               builtin-mccs+glpk
# install-criteria     -removed,-count[avoid-version,changed],-count[version-lag,request],-count[version-lag,changed],-count[missing-depexts,changed],-changed
# upgrade-criteria     -removed,-count[avoid-version,changed],-count[version-lag,solution],-count[missing-depexts,changed],-new
# jobs                 3
# repositories         1 (http) (default repo at 0eacced0)
# pinned               2 (git)
# current-switch       default
# ocaml:native         true
# ocaml:native-tools   true
# ocaml:native-dynlink true
# ocaml:stubsdir       /usr/lib/ocaml/stublibs:/usr/lib/ocaml
# ocaml:preinstalled   true
# ocaml:compiler       system
# ocaml-system:path    /usr/bin

[rjbou]

pacman shouldn't take so much time to query for depexts. Can you launch opam list --installed --debug to find where is the bottleneck? It have timestamps.

[Willenbrink]

I'm using openSUSE Tumbleweed and opam 2.1.1 and ran into the same issue. Interestingly, loading the opam files from repo default takes about 7s but deleting this folder also takes 1.5s which seems weird to me (or some long operation after deletion is missing a timestamp).

opam config report

> opam config report
# opam config report
# opam-version         2.1.1 
# self-upgrade         no
# system               arch=x86_64 os=linux os-distribution=opensuse-tumbleweed os-version=20211117
# solver               builtin-mccs+glpk
# install-criteria     -removed,-count[avoid-version,changed],-count[version-lag,request],-count[version-lag,changed],-count[missing-depexts,changed],-changed
# upgrade-criteria     -removed,-count[avoid-version,changed],-count[version-lag,solution],-count[missing-depexts,changed],-new
# jobs                 3
# repositories         1 (http), 1 (version-controlled) (default repo at c7b1bb0d)
# pinned               1 (git)
# current-switch       4.12.0+domains
# ocaml:native         true
# ocaml:native-tools   true
# ocaml:native-dynlink true
# ocaml:stubsdir       /home/sewi/.opam/4.12.0+domains/lib/ocaml/stublibs:/home/sewi/.opam/4.12.0+domains/lib/ocaml
# ocaml:preinstalled   false
# ocaml:compiler       4.12.0+domains+nnp

opam list --installed --debug

> opam list --installed --debug
00:00.013  CLI                    Parsing CLI version 2.1
00:00.013  GSTATE                 LOAD-GLOBAL-STATE @ /home/sewi/.opam
00:00.013  RSTATE                 LOAD-REPOSITORY-STATE @ /home/sewi/.opam
00:00.013  RSTATE                 No cache found
00:00.013  SYSTEM                 mkdir /tmp/opam-5718-ed05fb
00:07.918  RSTATE                 loaded opam files from repo default in 6.697s
00:08.040  RSTATE                 loaded opam files from repo multicore in 0.041s
00:08.040  CACHE(repository)      Writing the repository cache to ~/.opam/repo/state-3F983742.cache ...
00:08.386  CACHE(repository)      ~/.opam/repo/state-3F983742.cache written in 0.346s
00:08.386  STATE                  LOAD-SWITCH-STATE @ 4.12.0+domains
00:08.442  CACHE(installed)       Writing the installed cache to ~/.opam/4.12.0+domains/.opam-switch/packages/cache ...
00:08.443  CACHE(installed)       ~/.opam/4.12.0+domains/.opam-switch/packages/cache written in 0.035s
00:08.496  STATE                  Switch state loaded in 0.110s
# Packages matching: installed
# Name              # Installed    # Synopsis
astring             0.8.5          Alternative String module for OCaml
base                v0.14.1        Full standard library replacement for OCaml
base-bigarray       base
base-bytes          base           Bytes library distributed with the OCaml compiler
base-domains        base
base-threads        base
base-unix           base
bigarray-compat     1.0.0          Compatibility library to use Stdlib.Bigarray when possible
biniou              1.2.1          Binary data format designed for speed, safety, ease of use and backward compatibi
bogue               20210917       GUI library for ocaml, with animations, based on SDL2
bos                 0.2.1          Basic OS interaction for OCaml
camomile            1.0.2          A Unicode library
charInfo_width      1.1.0          Determine column width for a character
cmdliner            1.0.4          Declarative definition of command line interfaces for OCaml
conf-emacs          1              Virtual package to install the Emacs editor
conf-libffi         2.0.0          Virtual package relying on libffi system installation
conf-libxcursor     1              Virtual package relying on an libXcursor system installation
conf-libxi          1              Virtual package relying on an libXi system installation
conf-libxinerama    1              Virtual package relying on an libXinerama system installation
conf-libxrandr      1              Virtual package relying on an libXRandR system installation
conf-mesa           1              Virtual package relying on an mesa system installation
conf-pkg-config     2              Check if pkg-config is installed and create an opam switch local pkgconfig folder
conf-sdl2           1              Virtual package relying on a SDL2 system installation
conf-sdl2-image     1              Virtual package relying on a sdl2-image system installation
conf-sdl2-ttf       1              Virtual package relying on a sdl2-ttf system installation
containers          3.6.1          A modular, clean and powerful extension of the OCaml standard library
cppo                1.6.8          Code preprocessor like cpp for OCaml
csexp               1.5.1          Parsing and printing of S-expressions in Canonical form
ctypes              0.19.1         Combinators for binding to C libraries without writing any C
ctypes-foreign      0.18.0         Virtual package for enabling the ctypes.foreign subpackage.
dot-merlin-reader   4.1            Reads config files for merlin
dune                2.9.1          Fast, portable, and opinionated build system
dune-build-info     2.9.1          Embed build informations inside executable
dune-configurator   2.9.1          Helper library for gathering system configuration
easy-format         1.3.2          High-level and functional interface to the Format module of the OCaml standard li
either              1.0.0          Compatibility Either module
fix                 20201120       Facilities for memoization and fixed points
fmt                 0.9.0          OCaml Format pretty-printer combinators
fpath               0.7.3          File system paths for OCaml
gg                  0.9.3          Basic types for computer graphics in OCaml
integers            0.5.1          Various signed and unsigned integer types for OCaml
jane-street-headers v0.14.0        Jane Street C header files
jst-config          v0.14.1        Compile-time configuration for Jane Street libraries
lambda-term         3.1.0          Terminal manipulation library for OCaml
landmarks           1.4            A simple profiling library
landmarks-ppx       1.4            Preprocessor instrumenting code using the landmarks library
logs                0.7.0          Logging infrastructure for OCaml
lwt                 5.4.2          Promises and event-driven I/O
lwt_log             1.1.1          Lwt logging library (deprecated)
lwt_react           1.1.4          Helpers for using React with Lwt
menhir              20211012       An LR(1) parser generator
menhirLib           20211012       Runtime support library for parsers generated by Menhir
menhirSdk           20211012       Compile-time library for auxiliary tools related to Menhir
merlin              4.3.1-412      Editor helper, provides completion, typing and source browsing in Vim and Emacs
mew                 0.1.0          Modal editing witch
mew_vi              0.5.0          Modal editing witch, VI interpreter
mmap                1.1.0          File mapping functionality
num                 1.4            The legacy Num library for arbitrary-precision integer and rational arithmetic
ocaml               4.12.0         The OCaml compiler (virtual package)
ocaml-compiler-libs v0.12.4        OCaml compiler libraries repackaged
ocaml-config        2              OCaml Switch Configuration
ocaml-option-nnp    1              Set OCaml to be compiled with --disable-naked-pointers
ocaml-print-intf    1.2.0          Display human-readable OCaml interface from a compiled .cmi
ocaml-variants      4.12.0+domains OCaml 4.12.0, with support for multicore domains
ocamlbuild          0.14.0         OCamlbuild is a build system with builtin rules to easily build most OCaml projec
ocamlfind           1.9.1          A library manager for OCaml
ocamlformat         0.19.0         Auto-formatter for OCaml code
ocp-indent          1.8.1          A simple tool to indent OCaml programs
ocplib-endian       1.1            Optimised functions to read and write int16/32/64 from strings and bigarrays
octavius            1.2.2          Ocamldoc comment syntax parser
odoc-parser         0.9.0          Parser for ocaml documentation comments
ppx_assert          v0.14.0        Assert-like extension nodes that raise useful errors on failure
ppx_base            v0.14.0        Base set of ppx rewriters
ppx_cold            v0.14.0        Expands [@cold] into [@inline never][@specialise never][@local never]
ppx_compare         v0.14.0        Generation of comparison functions from types
ppx_cstubs          0.6.1.2        Preprocessor for easier stub generation with ctypes
ppx_derivers        1.2.1          Shared [@@deriving] plugin registry
ppx_deriving        5.2.1          Type-driven code generation for OCaml
ppx_effects         ~dev           pinned to version ~dev at git+https://github.com/CraigFe/ppx_effects.git
ppx_enumerate       v0.14.0        Generate a list containing all values of a finite type
ppx_expect          v0.14.1        Cram like framework for OCaml
ppx_hash            v0.14.0        A ppx rewriter that generates hash functions from type expressions and definition
ppx_here            v0.14.0        Expands [%here] into its location
ppx_inline_test     v0.14.1        Syntax extension for writing in-line tests in ocaml code
ppx_js_style        v0.14.1        Code style checker for Jane Street Packages
ppx_optcomp         v0.14.3        Optional compilation for OCaml
ppx_sexp_conv       v0.14.3        [@@deriving] plugin to generate S-expression conversion functions
ppx_yojson_conv     v0.14.0        [@@deriving] plugin to generate Yojson conversion functions
ppx_yojson_conv_lib v0.14.0        Runtime lib for ppx_yojson_conv
ppxlib              0.23.0         Standard library for ppx rewriters
raylib              0.4.0          OCaml bindings for raylib
re                  1.10.3         RE is a regular expression library for OCaml
react               1.2.1          Declarative events and signals for OCaml
result              1.5            Compatibility Result module
rresult             0.7.0          Result value combinators for OCaml
seq                 base           Compatibility package for OCaml's standard iterator type starting from 4.07.
sexplib0            v0.14.0        Library containing the definition of S-expressions and some base converters
stdio               v0.14.0        Standard IO library for OCaml
stdlib-shims        0.3.0          Backport some of the new stdlib features to older compiler
tgls                0.8.5          Thin bindings to OpenGL {3,4} and OpenGL ES {2,3} for OCaml
time_now            v0.14.0        Reports the current time
topkg               1.0.4          The transitory OCaml software packager
trie                1.0.0          Strict impure trie tree
tsdl                0.9.8          Thin bindings to SDL for OCaml
tsdl-image          0.3.0          SDL2_Image bindings to go with Tsdl
tsdl-ttf            0.3            SDL2_Ttf bindings to go with Tsdl
tuareg              2.2.0          OCaml mode for GNU Emacs and XEmacs.
uchar               0.0.2          Compatibility library for OCaml's Uchar module
user-setup          0.7            Helper for the configuration of editors for the use of OCaml tools
utop                2.8.0          Universal toplevel for OCaml
uucp                14.0.0         Unicode character properties for OCaml
uuseg               14.0.0         Unicode text segmentation for OCaml
uutf                1.0.2          Non-blocking streaming Unicode codec for OCaml
yojson              1.7.0          Yojson is an optimized parsing and printing library for the JSON format
zed                 3.1.0          Abstract engine for text edition in OCaml
00:08.500  SYSTEM                 rmdir /tmp/opam-5718-ed05fb/multicore
00:08.512  SYSTEM                 rmdir /tmp/opam-5718-ed05fb/default
00:09.954  SYSTEM                 rmdir /tmp/opam-5718-ed05fb

[rjbou]

related to #4586

[gadmm]

Same issue for the startup time in Emacs, narrowed down to a call to opam list --installed --short --safe --color=never in opam-user-setup.el. However, opam version is 2.0.8. The slowness appeared after upgrading from Ubuntu 20.10 to 21.04. Installing the non-installed package from the list (ocp-index) (as suggested in #4875) made it much faster.