Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

virtualenv fails sometimes when run concurrently #1824

Closed
gozdal opened this issue May 12, 2020 · 3 comments · Fixed by #1826
Closed

virtualenv fails sometimes when run concurrently #1824

gozdal opened this issue May 12, 2020 · 3 comments · Fixed by #1826
Labels
bug

Comments

@gozdal
Copy link

gozdal commented May 12, 2020

Issue

Sometimes our CI builds fail when creating virtualenv. We've tracked the problem to builds running in parallel, trying to create a virtualenv in different directories at the same time.
Able to reproduce it with a simple test:

rm -rf *; parallel python3.7 -m virtualenv  ::: $(seq 10)

WARNING: run in a scratch directory (note rm -rf *).

Most of the times the test passes, but sometimes it fails with errors like these:

$ parallel python3.7 -m virtualenv --clear ::: $(seq 10)
created virtual environment CPython3.7.7.final.0-64 in 169ms
  creator CPython3Posix(dest=/home/jenkins/virtualenv/2, clear=True, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/jenkins/.local/share/virtualenv/seed-app-data/v1.0.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
Exception in thread Thread-5:
Traceback (most recent call last):
  File "/opt/starfish/python3.7-shared/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/opt/starfish/python3.7-shared/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/starfish/python3.7-shared/lib/python3.7/site-packages/virtualenv/seed/via_app_data/via_app_data.py", line 50, in _install
    installer.build_image()
  File "/opt/starfish/python3.7-shared/lib/python3.7/site-packages/virtualenv/seed/via_app_data/pip_install/base.py", line 59, in build_image
    zip_ref.extractall(str(self._image_dir))
  File "/opt/starfish/python3.7-shared/lib/python3.7/zipfile.py", line 1636, in extractall
    self._extract_member(zipinfo, path, pwd)
  File "/opt/starfish/python3.7-shared/lib/python3.7/zipfile.py", line 1690, in _extract_member
    open(targetpath, "wb") as target:
FileNotFoundError: [Errno 2] No such file or directory: '/home/jenkins/.local/share/virtualenv/seed-app-data/v1.0.1/3.7/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel-0.34.2.dist-info/RECORD'

Exception in thread Thread-6:
Traceback (most recent call last):
  File "/opt/starfish/python3.7-shared/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/opt/starfish/python3.7-shared/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/starfish/python3.7-shared/lib/python3.7/site-packages/virtualenv/seed/via_app_data/via_app_data.py", line 50, in _install
    installer.build_image()
  File "/opt/starfish/python3.7-shared/lib/python3.7/site-packages/virtualenv/seed/via_app_data/pip_install/base.py", line 59, in build_image
    zip_ref.extractall(str(self._image_dir))
  File "/opt/starfish/python3.7-shared/lib/python3.7/zipfile.py", line 1636, in extractall
    self._extract_member(zipinfo, path, pwd)
  File "/opt/starfish/python3.7-shared/lib/python3.7/zipfile.py", line 1682, in _extract_member
    os.makedirs(upperdirs)
  File "/opt/starfish/python3.7-shared/lib/python3.7/os.py", line 223, in makedirs
    mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/home/jenkins/.local/share/virtualenv/seed-app-data/v1.0.1/3.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/pip/_internal/cli'

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/opt/starfish/python3.7-shared/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/opt/starfish/python3.7-shared/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/starfish/python3.7-shared/lib/python3.7/site-packages/virtualenv/seed/via_app_data/via_app_data.py", line 50, in _install
    installer.build_image()
  File "/opt/starfish/python3.7-shared/lib/python3.7/site-packages/virtualenv/seed/via_app_data/pip_install/base.py", line 59, in build_image
    zip_ref.extractall(str(self._image_dir))
  File "/opt/starfish/python3.7-shared/lib/python3.7/zipfile.py", line 1636, in extractall
    self._extract_member(zipinfo, path, pwd)
  File "/opt/starfish/python3.7-shared/lib/python3.7/zipfile.py", line 1682, in _extract_member
    os.makedirs(upperdirs)
  File "/opt/starfish/python3.7-shared/lib/python3.7/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/opt/starfish/python3.7-shared/lib/python3.7/os.py", line 223, in makedirs
    mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/home/jenkins/.local/share/virtualenv/seed-app-data/v1.0.1/3.7/image/CopyPipInstall/setuptools-46.1.3-py3-none-any/pkg_resources/_vendor'

created virtual environment CPython3.7.7.final.0-64 in 151ms
  creator CPython3Posix(dest=/home/jenkins/virtualenv/1, clear=True, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/jenkins/.local/share/virtualenv/seed-app-data/v1.0.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/opt/starfish/python3.7-shared/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/opt/starfish/python3.7-shared/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/starfish/python3.7-shared/lib/python3.7/site-packages/virtualenv/seed/via_app_data/via_app_data.py", line 75, in _get
    self.app_data,
  File "/opt/starfish/python3.7-shared/lib/python3.7/site-packages/virtualenv/seed/embed/wheels/acquire.py", line 28, in get_wheels
    acquire_from_bundle(processed, for_py_version, wheel_cache_dir)
  File "/opt/starfish/python3.7-shared/lib/python3.7/site-packages/virtualenv/seed/embed/wheels/acquire.py", line 57, in acquire_from_bundle
    copy2(str(bundle), str(bundled_wheel_file))
  File "/opt/starfish/python3.7-shared/lib/python3.7/shutil.py", line 267, in copy2
    copystat(src, dst, follow_symlinks=follow_symlinks)
  File "/opt/starfish/python3.7-shared/lib/python3.7/shutil.py", line 206, in copystat
    follow_symlinks=follow)
FileNotFoundError: [Errno 2] No such file or directory

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/opt/starfish/python3.7-shared/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/opt/starfish/python3.7-shared/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/starfish/python3.7-shared/lib/python3.7/site-packages/virtualenv/seed/via_app_data/via_app_data.py", line 75, in _get
    self.app_data,
  File "/opt/starfish/python3.7-shared/lib/python3.7/site-packages/virtualenv/seed/embed/wheels/acquire.py", line 28, in get_wheels
    acquire_from_bundle(processed, for_py_version, wheel_cache_dir)
  File "/opt/starfish/python3.7-shared/lib/python3.7/site-packages/virtualenv/seed/embed/wheels/acquire.py", line 57, in acquire_from_bundle
    copy2(str(bundle), str(bundled_wheel_file))
  File "/opt/starfish/python3.7-shared/lib/python3.7/shutil.py", line 267, in copy2
    copystat(src, dst, follow_symlinks=follow_symlinks)
  File "/opt/starfish/python3.7-shared/lib/python3.7/shutil.py", line 206, in copystat
    follow_symlinks=follow)
FileNotFoundError: [Errno 2] No such file or directory

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/opt/starfish/python3.7-shared/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/opt/starfish/python3.7-shared/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/starfish/python3.7-shared/lib/python3.7/site-packages/virtualenv/seed/via_app_data/via_app_data.py", line 50, in _install
    installer.build_image()
  File "/opt/starfish/python3.7-shared/lib/python3.7/site-packages/virtualenv/seed/via_app_data/pip_install/base.py", line 58, in build_image
    with zipfile.ZipFile(str(self._wheel)) as zip_ref:
  File "/opt/starfish/python3.7-shared/lib/python3.7/zipfile.py", line 1240, in __init__
    self.fp = io.open(file, filemode)
FileNotFoundError: [Errno 2] No such file or directory: '/home/jenkins/.local/share/virtualenv/seed-app-data/v1.0.1/3.7/wheels/wheel-0.34.2-py2.py3-none-any.whl'

created virtual environment CPython3.7.7.final.0-64 in 293ms
  creator CPython3Posix(dest=/home/jenkins/virtualenv/8, clear=True, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/jenkins/.local/share/virtualenv/seed-app-data/v1.0.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
created virtual environment CPython3.7.7.final.0-64 in 349ms
  creator CPython3Posix(dest=/home/jenkins/virtualenv/10, clear=True, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/jenkins/.local/share/virtualenv/seed-app-data/v1.0.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
created virtual environment CPython3.7.7.final.0-64 in 402ms
  creator CPython3Posix(dest=/home/jenkins/virtualenv/5, clear=True, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/jenkins/.local/share/virtualenv/seed-app-data/v1.0.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
created virtual environment CPython3.7.7.final.0-64 in 462ms
  creator CPython3Posix(dest=/home/jenkins/virtualenv/4, clear=True, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/jenkins/.local/share/virtualenv/seed-app-data/v1.0.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
created virtual environment CPython3.7.7.final.0-64 in 509ms
  creator CPython3Posix(dest=/home/jenkins/virtualenv/9, clear=True, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/jenkins/.local/share/virtualenv/seed-app-data/v1.0.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
created virtual environment CPython3.7.7.final.0-64 in 572ms
  creator CPython3Posix(dest=/home/jenkins/virtualenv/3, clear=True, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/jenkins/.local/share/virtualenv/seed-app-data/v1.0.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
created virtual environment CPython3.7.7.final.0-64 in 625ms
  creator CPython3Posix(dest=/home/jenkins/virtualenv/7, clear=True, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/jenkins/.local/share/virtualenv/seed-app-data/v1.0.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
created virtual environment CPython3.7.7.final.0-64 in 714ms
  creator CPython3Posix(dest=/home/jenkins/virtualenv/6, clear=True, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/jenkins/.local/share/virtualenv/seed-app-data/v1.0.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

Environment

Provide at least:

  • OS: Ubuntu 16, Ubuntu 18, CentOS 7
  • pip list of the host python where virtualenv is installed:
$ python3.7 -m pip freeze --all
appdirs==1.4.3
distlib==0.3.0
filelock==3.0.12
importlib-metadata==1.6.0
pip==20.0.2
setuptools==46.1.3
six==1.14.0
virtualenv==20.0.18
wheel==0.34.2
zipp==3.1.0

Output of the virtual environment creation

See above
@gozdal gozdal added the bug label May 12, 2020
@gaborbernat
Copy link
Contributor

gaborbernat commented May 12, 2020 •
edited
Loading

This feels like is file lock bug, in particular, the use case described by tox-dev/filelock#27 (comment)

PS. This is not the case.

@gaborbernat
Copy link
Contributor

Ok, so got to the bottom of this. Seems we're deleting the application data cache folder when --clear is passed, due to the following subtlety:

from argparse import ArgumentParser

parser = ArgumentParser()
parser.add_argument("--clear", action="store_true", default=False)
parser.add_argument("--clear-app-data", action="store_true", default=False)

opts = parser.parse_args(["--clear"])
print(opts)

# prints Namespace(clear=True, clear_app_data=False)

however:

from argparse import ArgumentParser

parser = ArgumentParser()
parser.add_argument("--clear-app-data", action="store_true", default=False)

opts = parser.parse_args(["--clear"])
print(opts)

# prints Namespace(clear_app_data=True)

😲😲😲😲😲🤦‍♂️🤦‍♂️🤦‍♂️🤦‍♂️🤦‍♂️

@gaborbernat gaborbernat mentioned this issue May 13, 2020
Merged
@gaborbernat
Copy link
Contributor

This now has been released via https://virtualenv.pypa.io/en/latest/changelog.html#v20-0-21-2020-05-20

@pypa pypa locked and limited conversation to collaborators Jan 14, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Rename clear-app-data to reset-app-data to avoid cli parser abbreviation bug gaborbernat/virtualenv
2 participants
@gozdal @gaborbernat