Unable to cache available package list in jupyter when mamba is installed #170

tlvu · 2022-01-14T18:33:41Z

Description

Since we added mamba into our conda env, we are unable to update the cache of available package list. The extension used to work properly before. I removed mamba and the extension works again !

{
   "command" : "/opt/conda/bin/mamba repoquery search * --json",
   "conda_info" : {
      "GID" : 1000,
      "UID" : 1000,
      "active_prefix" : null,
      "active_prefix_name" : null,
      "av_data_dir" : "/opt/conda/etc/conda",
      "av_metadata_url_base" : null,
      "channels" : [
         "https://repo.anaconda.com/pkgs/main/linux-64",
         "https://repo.anaconda.com/pkgs/main/noarch",
         "https://repo.anaconda.com/pkgs/r/linux-64",
         "https://repo.anaconda.com/pkgs/r/noarch"
      ],
      "conda_build_version" : "not installed",
      "conda_env_version" : "4.11.0",
      "conda_location" : "/opt/conda/lib/python3.9/site-packages/conda",
      "conda_prefix" : "/opt/conda",
      "conda_private" : false,
      "conda_shlvl" : -1,
      "conda_version" : "4.11.0",
      "config_files" : [],
      "default_prefix" : "/opt/conda",
      "env_vars" : {
         "CIO_TEST" : "<not set>",
         "CONDA_ROOT" : "/opt/conda",
         "CURL_CA_BUNDLE" : "<not set>",
         "PATH" : "/opt/conda/envs/birdy/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
         "REQUESTS_CA_BUNDLE" : "<not set>",
         "SSL_CERT_FILE" : "<not set>"
      },
      "envs" : [],
      "envs_dirs" : [
         "/opt/conda/envs",
         "/home/jenkins/.conda/envs"
      ],
      "netrc_file" : null,
      "offline" : false,
      "pkgs_dirs" : [
         "/opt/conda/pkgs",
         "/home/jenkins/.conda/pkgs"
      ],
      "platform" : "linux-64",
      "python_version" : "3.9.5.final.0",
      "rc_path" : "/home/jenkins/.condarc",
      "requests_version" : "2.26.0",
      "root_prefix" : "/opt/conda",
      "root_writable" : true,
      "site_dirs" : [],
      "sys.executable" : "/opt/conda/bin/python",
      "sys.prefix" : "/opt/conda",
      "sys.version" : "3.9.5 (default, Jun  4 2021, 12:28:51) \n[GCC 7.5.0]",
      "sys_rc_path" : "/opt/conda/.condarc",
      "user_agent" : "conda/4.11.0 requests/2.26.0 CPython/3.9.5 Linux/4.15.0-163-generic debian/10 glibc/2.28",
      "user_rc_path" : "/home/jenkins/.condarc",
      "virtual_pkgs" : [
         [
            "__linux",
            "4.15.0",
            "0"
         ],
         [
            "__glibc",
            "2.28",
            "0"
         ],
         [
            "__unix",
            "0",
            "0"
         ],
         [
            "__archspec",
            "1",
            "x86_64"
         ]
      ]
   },
   "error" : "RuntimeError('LockFile error. Aborting.')",
   "exception_name" : "RuntimeError",
   "exception_type" : "<class 'RuntimeError'>",
   "traceback" : "Traceback (most recent call last):\n  File \"/opt/conda/lib/python3.9/site-packages/conda/exceptions.py\", line 1080, in __call__\n    return func(*args, **kwargs)\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/mamba.py\", line 917, in exception_converter\n    raise e\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/mamba.py\", line 911, in exception_converter\n    exit_code = _wrapped_main(*args, **kwargs)\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/mamba.py\", line 869, in _wrapped_main\n    result = do_call(args, p)\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/mamba.py\", line 744, in do_call\n    exit_code = repoquery(args, parser)\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/mamba.py\", line 686, in repoquery\n    pool = repoquery_api.create_pool(channels, platform, use_installed)\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/repoquery.py\", line 47, in create_pool\n    load_channels(\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/utils.py\", line 122, in load_channels\n    index = get_index(\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/utils.py\", line 103, in get_index\n    is_downloaded = dlist.download(True)\nRuntimeError: LockFile error. Aborting.\n"
}

Reproduce

Go to '...'
Click on '...'
Scroll down to '...'
See error '...'

Expected behavior

Context

Python package version: Python 3.7.12
Extension version: mamba_gator 5.1.2 pyhd8ed1ab_0 conda-forge
General information:

$ conda run -n birdy jupyter labextension list
JupyterLab v3.2.5
/opt/conda/envs/birdy/share/jupyter/labextensions
        jupyter-leaflet v0.15.0 enabled OK
        jupyterlab-plotly v5.5.0 enabled OK
        jupyterlab-topbar-extension v0.6.1 enabled OK (python, jupyterlab-topbar)
        jupyterlab-jupytext v1.3.8 enabled OK (python, jupytext)
        jupyterlab-logout v0.5.0 enabled OK (python, jupyterlab-logout)
        jupyterlab-system-monitor v0.8.0 enabled OK (python, jupyterlab-system-monitor)
        nbdime-jupyterlab v2.1.1 enabled OK
        @mamba-org/gator-lab v3.0.2 enabled OK (python, mamba_gator)
        @jupyterlab/git v0.34.0 enabled OK (python, jupyterlab-git)
        @bokeh/jupyter_bokeh v3.0.4 enabled OK (python, jupyter_bokeh)
        @voila-dashboards/jupyterlab-preview v2.1.0 enabled OK (python, voila)
        @jupyter-server/resource-usage v0.5.0 enabled OK (python, jupyter-resource-usage)
        @pyviz/jupyterlab_pyviz v2.1.0 enabled OK (python, pyviz_comms)
        @jupyter-widgets/jupyterlab-manager v3.0.1 enabled OK (python, jupyterlab_widgets)
        @hadim/jupyter-archive v3.2.1 enabled OK (python, jupyter-archive)

Other labextensions (built into JupyterLab)
   app dir: /opt/conda/envs/birdy/share/jupyter/lab
        jupyterlab-dash v0.4.0 enabled OK


$ conda info

     active environment : base
    active env location : /opt/conda
            shell level : 1
       user config file : /root/.condarc
 populated config files : /root/.condarc
          conda version : 4.11.0
    conda-build version : not installed
         python version : 3.9.5.final.0
       virtual packages : __linux=4.15.0=0
                          __glibc=2.28=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /opt/conda  (writable)
      conda av data dir : /opt/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /opt/conda/pkgs
                          /root/.conda/pkgs
       envs directories : /opt/conda/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/4.11.0 requests/2.26.0 CPython/3.9.5 Linux/4.15.0-163-generic debian/10 glibc/2.28
                UID:GID : 0:0
             netrc file : None
           offline mode : False

Command Line Output

Paste the output from your command line running `jupyter lab` here, use `--debug` if possible.

Browser Output

Paste the output from your browser Javascript console here.

The text was updated successfully, but these errors were encountered:

tlvu · 2022-01-14T19:14:00Z

Mamba was installed in the base conda env while the Jupyter is installed in another conda env. Should mamba be installed in the same conda env as the Jupyter?

See mamba-org/gator#170.

fcollonval · 2022-02-08T07:54:47Z

Sorry for the late answer.

Mamba was installed in the base conda env while the Jupyter is installed in another conda env. Should mamba be installed in the same conda env as the Jupyter?

No it should not.

"error" : "RuntimeError('LockFile error. Aborting.')",

Could you try running the following command in a terminal on the machine?

/opt/conda/bin/mamba repoquery search * --json

tlvu · 2022-06-17T18:23:06Z

Could you try running the following command in a terminal on the machine?
/opt/conda/bin/mamba repoquery search * --json

@fcollonval Sorry for the late reply, from which dir I should run that command? From / I get this error:

usage: mamba [-h] [-V] command ...
mamba: error: unrecognized arguments: boot dev environment.yml etc home lib lib64 media mnt notebook_dir opt proc root run sbin srv sys tmp usr var

tlvu · 2022-06-17T18:53:42Z

Might be related, mamba install pyston_lite_autoload -c pyston -c conda-forge I get this error

    Traceback (most recent call last):
      File "/opt/conda/lib/python3.9/site-packages/conda/exceptions.py", line 1114, in __call__
        return func(*args, **kwargs)
      File "/opt/conda/lib/python3.9/site-packages/mamba/mamba.py", line 935, in exception_converter
        raise e
      File "/opt/conda/lib/python3.9/site-packages/mamba/mamba.py", line 929, in exception_converter
        exit_code = _wrapped_main(*args, **kwargs)
      File "/opt/conda/lib/python3.9/site-packages/mamba/mamba.py", line 887, in _wrapped_main
        result = do_call(args, p)
      File "/opt/conda/lib/python3.9/site-packages/mamba/mamba.py", line 750, in do_call
        exit_code = install(args, parser, "install")
      File "/opt/conda/lib/python3.9/site-packages/mamba/mamba.py", line 497, in install
        index = load_channels(pool, channels, repos)
      File "/opt/conda/lib/python3.9/site-packages/mamba/utils.py", line 129, in load_channels
        index = get_index(
      File "/opt/conda/lib/python3.9/site-packages/mamba/utils.py", line 110, in get_index
        is_downloaded = dlist.download(api.MAMBA_DOWNLOAD_FAILFAST)
    RuntimeError: Operation not permitted: '/opt/conda/pkgs/cache/47929eba.json'

But the same using conda works: conda install pyston_lite_autoload -c pyston -c conda-forge

conda-4.13.0
mamba-0.24.0

tlvu · 2022-06-17T18:59:52Z

@fcollonval I think I got what you wanted

$ /opt/conda/bin/mamba repoquery search birdy --json

{
  "command": "/opt/conda/bin/mamba repoquery search birdy --json",
  "conda_info": {
    "GID": 1000,
    "UID": 1000,
    "active_prefix": "/opt/conda/envs/birdy",
    "active_prefix_name": "birdy",
    "av_data_dir": "/opt/conda/etc/conda",
    "av_metadata_url_base": null,
    "channels": [
      "https://repo.anaconda.com/pkgs/main/linux-64",
      "https://repo.anaconda.com/pkgs/main/noarch",
      "https://repo.anaconda.com/pkgs/r/linux-64",
      "https://repo.anaconda.com/pkgs/r/noarch"
    ],
    "conda_build_version": "not installed",
    "conda_env_version": "4.13.0",
    "conda_location": "/opt/conda/lib/python3.9/site-packages/conda",
    "conda_prefix": "/opt/conda",
    "conda_private": false,
    "conda_shlvl": 2,
    "conda_version": "4.13.0",
    "config_files": [],
    "default_prefix": "/opt/conda/envs/birdy",
    "env_vars": {
      "CIO_TEST": "<not set>",
      "CONDA_DEFAULT_ENV": "birdy",
      "CONDA_EXE": "/opt/conda/bin/conda",
      "CONDA_PREFIX": "/opt/conda/envs/birdy",
      "CONDA_PREFIX_1": "/opt/conda",
      "CONDA_PROMPT_MODIFIER": "(birdy) ",
      "CONDA_PYTHON_EXE": "/opt/conda/bin/python",
      "CONDA_ROOT": "/opt/conda",
      "CONDA_SHLVL": "2",
      "CURL_CA_BUNDLE": "<not set>",
      "PATH": "/opt/conda/envs/birdy/bin:/opt/conda/condabin:/opt/conda/envs/birdy/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "REQUESTS_CA_BUNDLE": "<not set>",
      "SSL_CERT_FILE": "<not set>"
    },
    "envs": [],
    "envs_dirs": [
      "/opt/conda/envs",
      "/notebook_dir/writable-workspace/.home/.conda/envs"
    ],
    "netrc_file": null,
    "offline": false,
    "pkgs_dirs": [
      "/opt/conda/pkgs",
      "/notebook_dir/writable-workspace/.home/.conda/pkgs"
    ],
    "platform": "linux-64",
    "python_version": "3.9.12.final.0",
    "rc_path": "/notebook_dir/writable-workspace/.home/.condarc",
    "requests_version": "2.27.1",
    "root_prefix": "/opt/conda",
    "root_writable": true,
    "site_dirs": [],
    "sys.executable": "/opt/conda/bin/python",
    "sys.prefix": "/opt/conda",
    "sys.version": "3.9.12 (main, Apr  5 2022, 06:56:58) \n[GCC 7.5.0]",
    "sys_rc_path": "/opt/conda/.condarc",
    "user_agent": "conda/4.13.0 requests/2.27.1 CPython/3.9.12 Linux/4.18.0-348.20.1.el8_5.x86_64 debian/11 glibc/2.31",
    "user_rc_path": "/notebook_dir/writable-workspace/.home/.condarc",
    "virtual_pkgs": [
      [
        "__linux",
        "4.18.0",
        "0"
      ],
      [
        "__glibc",
        "2.31",
        "0"
      ],
      [
        "__unix",
        "0",
        "0"
      ],
      [
        "__archspec",
        "1",
        "x86_64"
      ]
    ]
  },
  "error": "RuntimeError(\"Operation not permitted: '/opt/conda/pkgs/cache/3e39a7aa.json'\")",
  "exception_name": "RuntimeError",
  "exception_type": "<class 'RuntimeError'>",
  "traceback": "Traceback (most recent call last):\n  File \"/opt/conda/lib/python3.9/site-packages/conda/exceptions.py\", line 1114, in __call__\n    return func(*args, **kwargs)\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/mamba.py\", line 935, in exception_converter\n    raise e\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/mamba.py\", line 929, in exception_converter\n    exit_code = _wrapped_main(*args, **kwargs)\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/mamba.py\", line 887, in _wrapped_main\n    result = do_call(args, p)\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/mamba.py\", line 760, in do_call\n    exit_code = repoquery(args, parser)\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/mamba.py\", line 702, in repoquery\n    pool = repoquery_api.create_pool(channels, platform, use_installed)\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/repoquery.py\", line 46, in create_pool\n    load_channels(\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/utils.py\", line 129, in load_channels\n    index = get_index(\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/utils.py\", line 110, in get_index\n    is_downloaded = dlist.download(api.MAMBA_DOWNLOAD_FAILFAST)\nRuntimeError: Operation not permitted: '/opt/conda/pkgs/cache/3e39a7aa.json'\n"
}

tlvu · 2022-06-17T19:05:47Z

"error": "RuntimeError("Operation not permitted: '/opt/conda/pkgs/cache/3e39a7aa.json'")",

But I do have write access to that file, I tried echo >> /opt/conda/pkgs/cache/3e39a7aa.json and it works fine.

…alled User `jenkins` had to own the folder `/opt/conda/pkgs/cache`. Just having write permissions there is not enough ! To fix this error: ``` $ mamba install pyston_lite_autoload -c pyston -c conda-forge Traceback (most recent call last): File "/opt/conda/lib/python3.9/site-packages/conda/exceptions.py", line 1114, in __call__ return func(*args, **kwargs) File "/opt/conda/lib/python3.9/site-packages/mamba/mamba.py", line 935, in exception_converter raise e File "/opt/conda/lib/python3.9/site-packages/mamba/mamba.py", line 929, in exception_converter exit_code = _wrapped_main(*args, **kwargs) File "/opt/conda/lib/python3.9/site-packages/mamba/mamba.py", line 887, in _wrapped_main result = do_call(args, p) File "/opt/conda/lib/python3.9/site-packages/mamba/mamba.py", line 750, in do_call exit_code = install(args, parser, "install") File "/opt/conda/lib/python3.9/site-packages/mamba/mamba.py", line 497, in install index = load_channels(pool, channels, repos) File "/opt/conda/lib/python3.9/site-packages/mamba/utils.py", line 129, in load_channels index = get_index( File "/opt/conda/lib/python3.9/site-packages/mamba/utils.py", line 110, in get_index is_downloaded = dlist.download(api.MAMBA_DOWNLOAD_FAILFAST) RuntimeError: Operation not permitted: '/opt/conda/pkgs/cache/47929eba.json' ``` See issue mamba-org/gator#170

tlvu · 2022-06-17T19:30:18Z

@fcollonval I found it, the current user have to be the owner of /opt/conda/pkgs/cache/ folder. Just having write-access is not enough for mamba. I was enough for conda.

Should I still keep this issue open to fix the "current user have to be the owner of /opt/conda/pkgs/cache/ folder" to be equivalent with conda? If this won't be fixed then I can close this issue.

fcollonval · 2022-06-18T17:10:30Z

Thanks a lot @tlvu to dig that one up. Would you mind opening an issue in the mamba repository?

tlvu · 2022-06-20T14:28:59Z

Thanks a lot @tlvu to dig that one up. Would you mind opening an issue in the mamba repository?

Sure. Will close this issue once the other one in mamba repo opened.

fcollonval · 2022-06-28T12:50:07Z

Thanks

pgramme · 2023-01-23T16:00:58Z

Sure. Will close this issue once the other one in mamba repo opened.

Hi! The issue already existed in the main mamba repo: mamba-org/mamba#488
To me it was fixed in mamba 1.2.0 (cf issue's discussion)

PS: I'm using mamba but not gator, Googling for the error first brought me here...

tlvu · 2023-03-03T16:32:40Z

Sure. Will close this issue once the other one in mamba repo opened.

Hi! The issue already existed in the main mamba repo: mamba-org/mamba#488 To me it was fixed in mamba 1.2.0 (cf issue's discussion)

PS: I'm using mamba but not gator, Googling for the error first brought me here...

Closing as per comment above. Thanks @pgramme

tlvu mentioned this issue Jan 14, 2022

docker: prevent manual pin of dependencies and improve build speed Ouranosinc/PAVICS-e2e-workflow-tests#95

Merged

tlvu added a commit to Ouranosinc/PAVICS-e2e-workflow-tests that referenced this issue Jan 14, 2022

docker: remove mamba for mamba_gator plugin to work properly

107c402

See mamba-org/gator#170.

tlvu closed this as completed Mar 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to cache available package list in jupyter when mamba is installed #170

Unable to cache available package list in jupyter when mamba is installed #170

tlvu commented Jan 14, 2022 •

edited

Loading

tlvu commented Jan 14, 2022 •

edited

Loading

fcollonval commented Feb 8, 2022

tlvu commented Jun 17, 2022

tlvu commented Jun 17, 2022

tlvu commented Jun 17, 2022

tlvu commented Jun 17, 2022

tlvu commented Jun 17, 2022

fcollonval commented Jun 18, 2022

tlvu commented Jun 20, 2022

fcollonval commented Jun 28, 2022

pgramme commented Jan 23, 2023

tlvu commented Mar 3, 2023

Unable to cache available package list in jupyter when mamba is installed #170

Unable to cache available package list in jupyter when mamba is installed #170

Comments

tlvu commented Jan 14, 2022 • edited Loading

Description

Reproduce

Expected behavior

Context

tlvu commented Jan 14, 2022 • edited Loading

fcollonval commented Feb 8, 2022

tlvu commented Jun 17, 2022

tlvu commented Jun 17, 2022

tlvu commented Jun 17, 2022

tlvu commented Jun 17, 2022

tlvu commented Jun 17, 2022

fcollonval commented Jun 18, 2022

tlvu commented Jun 20, 2022

fcollonval commented Jun 28, 2022

pgramme commented Jan 23, 2023

tlvu commented Mar 3, 2023

tlvu commented Jan 14, 2022 •

edited

Loading

tlvu commented Jan 14, 2022 •

edited

Loading