Why I'm not installing Pyleoclim yet

[jypeter]

@CommonClimate one of our LSCE users (@oliviermarti) asked me if I had pyleoclim installed in my multi-user python environment and I googled it and checked if I could safely install it. As it happens, there is no rush but you may still want some non-installation feedback (sounds weird!). We don't use this package yet, but many thanks for providing it!

• I'm trying to maintain a stable enough multi-user python environment with lots of different packages and I try to avoid installing pip stuff, because I'm always afraid I will break something. When you are not a pip specialist (my case), you don't really know what pip will do and if you will have a chance to go back. So it would be nice if pyleoclim could be installed with conda instead of pip. Note I actually have to use mamba instead of conda, in order to have lots of different packages in the same environment

• I had a look at the Installing Pyleoclim instructions, and the demjson-related notes had me worried. Well demjson is available on conda-forge (nice!), but is there a way you can use the built-in json package instead of something that has been inactive for 6 years and that apparently may cause problems?

• Before starting all of this, I had a look at the Pyleoclim pypi page. There are some broken links that somebody may want to fix:

  • Project links => Homepage: broken (https://github.com/LinkedEarth/Pyleoclim_util/pyleoclim)
  • underlying data structures: broken (https://pyleoclim-util.readthedocs.io/en/stable/core/ui.html)
  • Installation => package documentation: broken (https://pyleoclim-util.readthedocs.io/en/stable/installation.html)

[CommonClimate]

Thank you for the feedback @jypeter ! It is helpful. Let me try to address your points in turn:

• we have not previously considered the use case of a multi-user environment. However, if you work within conda, it should be easy to create a dedicated Python environment for pyleoclim. Is that not an option for you/your users? Note that pyleoclim plays nice with a host of packages (NumPy, SciPy, Matplotlib, pandas, Xarray, seaborn, scikit-learn, etc). These cover 95% of what climate scientists ever want to do, and it's quite rare that we come across conflicts with other workflows. It can happen, but it is rare... Maybe give it a try and see what happens? I guarantee it won't vaporize your computer.
• I am not sure why we ended up using pip, not conda; @fzhu2e should be able to tell you.
• the demjson issue is rather frustrating for us too. It has to do with the antiquated state of the LiPD utilities, which we have been trying to overhaul for a while. Due to administrative mishaps, the work could not be started until this month, but we now have a programmer on the case (@varunratnakar) and within a few months we will have brand new LiPD utilities that should be a lot more reliable, and work with modern packages. Please hold tight!
• thank you for documenting the broken links. @khider is preparing another release and she can hopefully fix those then.

We hope we can remove obstacles to installation soon!

[jypeter]

• I'm trying to maintain a multi-user and multi-packages environment, with all the packages that users request in the same environment, because it's hard to know which combination of packages a specific user will need at the same time. That's why I'd rather rely on conda than on pip for installation, because conda tells me what it will need to add/upgrade/downgrade and I can choose to go ahead or to cancel (whereas pip will just proceed). As a matter of fact, conda can't deal with complex environments, that's why I have to rely on mamba
• as I said, there is no rush, but as stated above, I'd rather rely on conda/mamba's dependencies handling
• it's nice if somebody can work on removing the demjson dependency!

You can ping me when you have something you can install with conda, and I will give it a try and tell you about the result

[CommonClimate]

Update: the demjson dependency was replaced by a demjson3 one in the latest version (0.7.3), getting rid of the previous issue.

Also, note that there is nothing magical about conda, and nothing toxic about pip. Indeed, the most popular machine learning library (scikit-learn) is only installable through pip. conda imposes more restrictions on package developers, which is why it is not in the cards for us right now given our small team and very active development.

Finally, if you need environments that are locked-in and create no conflicts, look at containerization, e.g. Docker: https://www.docker.com/blog/containerized-python-development-part-1/

[khider]

We also have ready to use containers here: https://quay.io/repository/2i2c/paleohack-2021?tab=tags

[jypeter]

What I don't like about pip is that there does not seem to be a dry-run option, but maybe someone knows more about this (I confess I have not spent much time investigating). When I install something in my existing environment, I want to know what's going to be upgraded/downgraded/installed before the actual installation

Note that scikit-lean can be installed from conda-forge. See transcript below. I had an old version, and I have easily updated it, after checking that it would not require changes to other packages!

>conda list | grep scikit
scikit-image              0.18.1           py38h51da96c_0    conda-forge
scikit-learn              0.24.1           py38h658cfdd_0    conda-forge

> mamba update scikit-learn scikit-image

                  __    __    __    __
                 /  \  /  \  /  \  /  \
                /    \/    \/    \/    \
███████████████/  /██/  /██/  /██/  /████████████████████████
              /  / \   / \   / \   / \  \____
             /  /   \_/   \_/   \_/   \    o \__,
            / _/                       \_____/  `
            |/
        ███╗   ███╗ █████╗ ███╗   ███╗██████╗  █████╗
        ████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗
        ██╔████╔██║███████║██╔████╔██║██████╔╝███████║
        ██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║
        ██║ ╚═╝ ██║██║  ██║██║ ╚═╝ ██║██████╔╝██║  ██║
        ╚═╝     ╚═╝╚═╝  ╚═╝╚═╝     ╚═╝╚═════╝ ╚═╝  ╚═╝

        mamba (0.22.1) supported by @QuantStack

        GitHub:  https://github.com/mamba-org/mamba
        Twitter: https://twitter.com/QuantStack

█████████████████████████████████████████████████████████████


Looking for: ['scikit-learn', 'scikit-image']

conda-forge/linux-64                                        Using cache
conda-forge/noarch                                          Using cache
pkgs/main/linux-64                                            No change
pkgs/main/noarch                                              No change
pkgs/r/noarch                                                 No change
pkgs/r/linux-64                                               No change

Pinned packages:
  - python 3.8.*


Transaction

  Prefix: /home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm_py3

  Updating specs:

   - scikit-learn
   - scikit-image
   - ca-certificates
   - certifi
   - openssl


  Package         Version  Build           Channel                    Size
────────────────────────────────────────────────────────────────────────────
  Upgrade:
────────────────────────────────────────────────────────────────────────────

  - libstdcxx-ng    9.3.0  h6de172a_18     installed
  + libstdcxx-ng   11.2.0  he4da1e4_14     conda-forge/linux-64     Cached
  - scikit-image   0.18.1  py38h51da96c_0  installed
  + scikit-image   0.19.2  py38h43a58ef_0  conda-forge/linux-64       13MB
  - scikit-learn   0.24.1  py38h658cfdd_0  installed
  + scikit-learn    1.0.2  py38h1561384_0  conda-forge/linux-64        8MB

  Summary:

  Upgrade: 3 packages

  Total download: 21MB

────────────────────────────────────────────────────────────────────────────

Confirm changes: [Y/n] Y
scikit-learn                                         8.4MB @  24.5MB/s  0.4s
scikit-image                                        12.7MB @  24.5MB/s  0.6s
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

> conda list | grep scikit
scikit-image              0.19.2           py38h43a58ef_0    conda-forge
scikit-learn              1.0.2            py38h1561384_0    conda-forge

[CommonClimate]

Hi @jypeter ,
pip does tell you which packages will be installed/upgraded/downgraded before you confirm installation, so you do get an idea of what will happen. However, I can see that you are still skittish about it, so perhaps a better option for your would be the research hub? Note that there are many institutions that manage custom Python environments for many users, without apparent problems.

Re: documentation bugs listed earlier, they should all be fixed in the new documentation. Please let us know if they are not.

Julien