python 3.7 environment using setup-python broken in 20200705.1

[ltalirz]

Describe the bug
Builds using the setup-python Github Action end up with "inconsistent" python environment, only for runners with virtual environment version 20200705.1 (20200625.0 was/is working fine).

Area for Triage:

Question, Bug, or Feature?:
Bug

Virtual environments affected

• macOS 10.15
• Ubuntu 16.04 LTS
• Ubuntu 18.04 LTS
• Ubuntu 20.04 LTS
• Windows Server 2016 R2
• Windows Server 2019

(only Ubuntu 18.04 tested)

Expected behavior
Uninstalling pyyaml package should work without errors.

Actual behavior

Uninstalling the pyyaml package results in

FileNotFoundError: [Errno 2] No such file or directory: '/opt/hostedtoolcache/Python/3.7.8/x64/lib/python3.7/site-packages/PyYAML-5.3.1-py3.7-linux-x86_64.egg'

Note: This has been observed only when requesting a python environment (3.7) that was not yet installed on the runner.
When requesting python 3.8 using setup-python, this issue does not occur (in this case, pip considers pyyaml not to be installed).

Failing run with 20200705.1 : https://github.com/aiidateam/aiida-core/pull/4208/checks?check_run_id=850446697#step:5:495
Successful run with 20200625.0: https://github.com/sphuber/aiida_core/runs/850405963#step:5:452
Successful run with 20200705.1 (after switching from python3.7 to python3.8 env): https://github.com/aiidateam/aiida-core/pull/4229/checks?check_run_id=850252814#step:5:452

P.S. I'm aware that I'm not providing a "minimum working example" here - I'm currently pressed for time.
I hope this can still help pinpoint the issue in case others experience the same (and I can add more information later if needed).

[maxim-lobanov]

@ltalirz , looks like between these image versions, Python was updated from 3.7.7 to 3.7.8. Probably, it could be the reason for errors.
Could you please try to specify exact version in your yaml to check if rollback to 3.7.7 works for you?

    - name: Set up Python 3.7
      uses: actions/setup-python@v1
      with:
        python-version: 3.7.7

[ltalirz]

Thanks, I've just tried this but I'm afraid this is not supported on the new image:

##[error]Version 3.7.7 with arch x64 not found
Available versions:

2.7.18 (x64)
3.5.9 (x64)
3.6.11 (x64)
3.7.8 (x64)
3.8.3 (x64)

https://github.com/ltalirz/aiida-core/runs/850587893

[maxim-lobanov]

Oh, sorry, it is my mistake, you should specify version v2 to install versions on-flight.

  - name: Set up Python 3.7
      uses: actions/setup-python@v2
      with:
        python-version: 3.7.7

[ltalirz]

No problem - I confirm that rolling back to python 3.7.7 resolves the issue
https://github.com/ltalirz/aiida-core/runs/850614175?check_suite_focus=true#step:5:123

So perhaps the reason I am the only one noticing this is that we were using an outdated version of the setup-python github action?
If I go back to installing version 3.7 (meaning 3.7.8) with v2 of the action, should the problem disappear?

[maxim-lobanov]

No, it won't resolve issue. The main feature of V2 is that it can install previous versions on-flight.

Looks like your code works correctly with Python 3.7.7 but fails with Python 3.7.8.
Could it be related to some updates / breaking changes / updated dependencies on Python side?

[ltalirz]

If my suspicion is correct, then simply calling "pip uninstall pyyaml" should fail.

This is what I get when I just do this:
https://github.com/ltalirz/aiida-core/runs/850664377?check_suite_focus=true#step:5:68

Edit: Ah, I need to make it non-interactive

[ltalirz]

Ok - I was wrong. Uninstalling pyyaml works just fine: https://github.com/ltalirz/aiida-core/runs/850675461?check_suite_focus=true#step:5:52

In this case, I suspect the issue must somehow be related to our dependencies. Thanks for your help!

[ltalirz]

Just in case others run into this: The issue is with the installation of the specific pyyaml version we're using (5.1.2):
https://github.com/ltalirz/aiida-core/runs/850699055?check_suite_focus=true#step:5:97

For some reason, this fails on the python 3.7.8 environment (but worked on the previous python 3.7.7 environment)'

Edit: The error only occurs, when uninstalling pyyaml 5.3.1 and installing pyyaml 5.1.2 in the same step.
Doing

        pip uninstall -y pyyaml
        pip install "pyyaml~=5.1.2"

works fine as well, see https://github.com/ltalirz/aiida-core/runs/858336584?check_suite_focus=true

[ltalirz]

Just mentioning that the issue with installing pyyaml 5.1.2 over an existing installation of pyyaml 5.3.1 does not seem to occur on the official python 3.7.8 docker containers.

I.e. the following runs fine:

docker pull python:3.7.8
docker run python:3.7.8 /bin/bash -c "python --version && pip install pyyaml==5.3.1 && pip install pyyaml==5.1.2"

Which may indicate that this is a specific issue of the virtual environment 20200705.1 ?

[vsafonkin]

Hi @ltalirz , looks like pyyaml 5.3.1 is installed during aws-sam-cli installation and it is not working properly, we are investigate how to resolve the issue.

[Borda]

The same also here - https://github.com/PyTorchLightning/pytorch-lightning/runs/889998403

EDIT: as #1202 (comment) corrected, only one env is failing

[maxim-lobanov]

@Borda , Only "pytest (ubuntu-18.04, 3.7, minimal)" job fails with the same issue in your build. All other jobs failed differently.

[DataGreed]

Is there any workaround for this? I have PyYAML==5.1.2 in my requirements.txt file and I run into this issue, too. What's even more annoying is that other jobs testing my project on other python versions got cancelled because this one fails.

[DataGreed]

Okay, in my case the error line read:

File "/opt/hostedtoolcache/Python/3.7.8/x64/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1731, in zipinfo
    return self._zip_manifests.load(self.loader.archive)
  File "/opt/hostedtoolcache/Python/3.7.8/x64/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1688, in load
    mtime = os.stat(path).st_mtime
FileNotFoundError: [Errno 2] No such file or directory: '/opt/hostedtoolcache/Python/3.7.8/x64/lib/python3.7/site-packages/PyYAML-5.3.1-py3.7-linux-x86_64.egg'
##[error]Process completed with exit code 1.

I had this in requirements.txt:

PyYAML==5.1.2

Changing it to:

PyYAML==5.3.1

Solved my problem.

Seems that one of the installed packages required PyYAML==5.3.1, but for some reason my requirements file had older version (actually it was generated by pip freeze, so I have no idea how it happened).

[Borda]

Well, is it general pip issue or just this CI, if so we do not force users to install a newer version just because if CI tests, right?

[miketimofeev]

@Borda @DataGreed we fixed the issue with python 3.7 a few days ago and started image deployment today. It usually takes 3-4 days to propagate to all the environments. Thank you for your patience.

[DataGreed]

@miketimofeev большое спасибо :)

[zgoda]

Is this new normal?

ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.
Successfully installed Babel-2.8.0 Bootstrap-Flask-1.4 Faker-4.1.1 Flask-1.1.2 Flask-Babel-1.0.0 Flask-Login-0.5.0 Flask-WTF-0.14.3 Jinja2-2.11.2 MarkupSafe-1.1.1 PeeWee-3.13.3 SecretStorage-3.1.2 WTForms-2.3.1 argon2-cffi-20.1.0 blinker-1.4 cffi-1.14.1 coverage-5.2.1 cryptography-3.0 decorator-4.4.2 factory-boy-2.12.0 inflection-0.5.0 iniconfig-1.0.0 itsdangerous-1.1.0 jeepney-0.4.3 keyring-21.2.1 keyrings.cryptfile-1.3.4 markdown-3.2.2 more-itertools-8.4.0 packaging-20.4 pluggy-0.13.1 py-1.9.0 pycparser-2.20 pycryptodome-3.9.8 pyparsing-2.4.7 pytest-6.0.0 pytest-cov-2.10.0 pytest-factoryboy-2.0.3 pytest-flask-1.0.0 pytest-mock-3.2.0 python-dateutil-2.8.1 python-dotenv-0.14.0 python-magic-0.4.18 sentry-sdk-0.16.2 text-unidecode-1.3 texttable-1.6.2 toml-0.10.1 validators-0.16.0

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

aws-sam-cli 1.0.0 requires Flask~=1.0.2, but you'll have flask 1.1.2 which is incompatible.
aws-sam-cli 1.0.0 requires python-dateutil<2.8.1,~=2.6, but you'll have python-dateutil 2.8.1 which is incompatible.

[miketimofeev]

@zgoda could you please provide the command, which produced such output?

[zgoda]

It's a log from one of my workflows running on Python 3.7 runtime, can't recall exactly which one but https://github.com/zgoda/bip/runs/929067312 has the same output.

Basically Python 3.7 runtime has a lot of either outdated or pre-release packages installed, like:

2020-07-30T18:56:03.2115932Z   Attempting uninstall: itsdangerous
2020-07-30T18:56:03.2171330Z     Found existing installation: itsdangerous 2.0.0a1
2020-07-30T18:56:03.2177836Z     Uninstalling itsdangerous-2.0.0a1:
2020-07-30T18:56:03.2617111Z       Successfully uninstalled itsdangerous-2.0.0a1
2020-07-30T18:56:03.2780445Z   Attempting uninstall: MarkupSafe
2020-07-30T18:56:03.2846347Z     Found existing installation: MarkupSafe 2.0.0a1
2020-07-30T18:56:03.2854577Z     Uninstalling MarkupSafe-2.0.0a1:
2020-07-30T18:56:03.3272337Z       Successfully uninstalled MarkupSafe-2.0.0a1
2020-07-30T18:56:03.3367490Z   Attempting uninstall: Jinja2
2020-07-30T18:56:03.3425437Z     Found existing installation: Jinja2 3.0.0a1
2020-07-30T18:56:03.3436269Z     Uninstalling Jinja2-3.0.0a1:
2020-07-30T18:56:03.3853792Z       Successfully uninstalled Jinja2-3.0.0a1

Flask 1.0.4 and python-dateutil 2.8.0 are installed with aws-sam-cli but anyway they should not be in a runtime used to run tests.

[miketimofeev]

@zgoda yeah, this is the same issue with aws-sam packages. It will be fixed on the next week's deployment.

[miketimofeev]

@ltalirz we've completed image deployment. Could you check everything is fine now, please?

[ltalirz]

Thanks @miketimofeev - looks to me like the issue is resolved!