virtual environment is failing

[systemlogic]

Thanks for submitting an issue!

If submitting a BUG please provide:

• OSError: [Errno 8] Exec format error:pip
• OS : ubuntu Pip list

Package                       Version               
----------------------------- ----------------------
appdirs                       1.4.3                 
apturl                        0.5.2                 
beautifulsoup4                4.4.1                 
blinker                       1.3                   
Brlapi                        0.6.4                 
certifi                       2019.11.28            
chardet                       2.3.0                 
checkbox-support              0.22                  
command-not-found             0.3                   
cryptography                  1.2.3                 
defer                         1.0.6                 
feedparser                    5.1.3                 
filelock                      3.0.12                
guacamole                     0.9.2                 
html5lib                      0.999                 
httplib2                      0.9.1                 
idna                          2.0                   
importlib-metadata            1.5.0                 
Jinja2                        2.8                   
language-selector             0.1                   
lightdm-gtk-greeter-settings  1.2.1                 
louis                         2.6.4                 
lxml                          3.5.0                 
Mako                          1.0.3                 
MarkupSafe                    0.23                  
oauthlib                      1.0.3                 
onboard                       1.2.0                 
padme                         1.1.1                 
pexpect                       4.0.1                 
Pillow                        3.1.2                 
pip                           20.0.2                
pipenv                        2018.5.18             
plainbox                      0.25                  
ptyprocess                    0.5                   
pyasn1                        0.1.9                 
pycups                        1.9.73                
pycurl                        7.43.0                
pygobject                     3.20.0                
PyJWT                         1.3.0                 
pyparsing                     2.0.3                 
python-apt                    1.1.0b1+ubuntu0.16.4.8
python-debian                 0.1.27                
python-systemd                231                   
pyxdg                         0.25                  
reportlab                     3.3.0                 
requests                      2.9.1                 
sessioninstaller              0.0.0                 
setuptools                    45.2.0                
six                           1.14.0                
ssh-import-id                 5.5                   
system-service                0.3                   
ubuntu-drivers-common         0.0.0                 
ufw                           0.35                  
unattended-upgrades           0.1                   
unity-scope-calculator        0.1                   
unity-scope-chromiumbookmarks 0.1                   
unity-scope-colourlovers      0.1                   
unity-scope-devhelp           0.1                   
unity-scope-firefoxbookmarks  0.1                   
unity-scope-gdrive            0.7                   
unity-scope-manpages          0.1                   
unity-scope-openclipart       0.1                   
unity-scope-texdoc            0.1                   
unity-scope-tomboy            0.1                   
unity-scope-virtualbox        0.1                   
unity-scope-yelp              0.1                   
unity-scope-zotero            0.1                   
urllib3                       1.13.1                
usb-creator                   0.3.0                 
virtualenv                    16.0.0                
virtualenv-clone              0.5.3                 
wheel                         0.29.0                
xdiagnose                     3.8.4.1               
xkit                          0.0.0                 
XlsxWriter                    0.7.3                 
zipp                          2.2.0 

if submitting an ENHANCEMENT issue:

• creating virtual environment is failing on ubuntu
• revert the changes made on 10th Feb
• if you have want help to do a PR yourself

[gaborbernat]

Thanks for opening an issue and sorry for the bug. Can you please attach the output of the virtual environment creation with -vvv?

[systemlogic]

@gaborbernat below target works on my MacOs but fails on ubuntu where all CI checks are running.

ERROR: /home/harora37/cognitive-data-platform/tools/dependencies/BUILD:17:1: Building custom python environment for repo_deps failed (Exit 1)
Virtualenv already exists!
Removing existing virtualenv…
Creating a virtualenv for this project…

OSError: [Errno 8] Exec format error: '/home/harora37/.cache/bazel/_bazel_harora37/4614b08a42facc1b3b190dce93d1ca57/execroot/com_ibm_monorepo/bazel-out/k8-fastbuild/bin/tools/dependencies/repo_deps_bzl_pipenv_interpreter.sh.runfiles/.venv/bin/pip'
Sync for generating 'bazel-out/k8-fastbuild/bin/tools/dependencies/repo_deps_bzl_pipenv_interpreter.sh' failed (exit code 1)

==========================================================================
File bazel-out/k8-fastbuild/bin/tools/dependencies/repo_deps_bzl_pipenv_interpreter.sh.runfiles/Pipfile.lock
--------------------------------------------------------------------------
     1	{
     2	    "_meta": {
     3	        "hash": {
     4	            "sha256": "69399f277c09eecf96616105c9916c1769cffe963eaae92c6b8688367d684808"
     5	        },
     6	        "pipfile-spec": 6,
     7	        "requires": {},
     8	        "sources": [
     9	            {
    10	                "name": "pypi",
    11	                "url": "https://harora37%40someDomain.com:somekey@some.artifactory.com/artifactory/api/pypi/cedp-pypi-virtual/simple",
    12	                "verify_ssl": true
    13	            }
    14	        ]
    15	    },
    16	    "default": {
    17	        "absl-py": {
    18	            "hashes": [
    19	                "sha256:75e737d6ce7723d9ff9b7aa1ba3233c34be62ef18d5859e706b8fdc828989830"
    20	            ],
    21	            "index": "pypi",
    22	            "version": "==0.9.0"
    23	        },
    24	        "beautifulsoup4": {
    25	            "hashes": [
    26	                "sha256:05fd825eb01c290877657a56df4c6e4c311b3965bda790c613a3d6fb01a5462a",
    27	                "sha256:9fbb4d6e48ecd30bcacc5b63b94088192dcda178513b2ae3c394229f8911b887",
    28	                "sha256:e1505eeed31b0f4ce2dbb3bc8eb256c04cc2b3b72af7d551a4ab6efd5cbe5dae"
    29	            ],
    30	            "version": "==4.8.2"
    31	        },
    32	        "google": {
    33	            "hashes": [
    34	                "sha256:c03517e6f9eec7a03c07b864f9a64384fd4bc1efce9034d42e3f90addc9fe4ee",
    35	                "sha256:ff7d997dbf33be666ed5deb8c5abf5d103c27ec199559b6d994923469e26bb48"
    36	            ],
    37	            "index": "pypi",
    38	            "version": "==2.0.3"
    39	        },
    40	        "prettytable": {
    41	            "hashes": [
    42	                "sha256:2d5460dc9db74a32bcc8f9f67de68b2c4f4d2f01fa3bd518764c69156d9cacd9",
    43	                "sha256:853c116513625c738dc3ce1aee148b5b5757a86727e67eff6502c7ca59d43c36",
    44	                "sha256:a53da3b43d7a5c229b5e3ca2892ef982c46b7923b51e98f0db49956531211c4f"
    45	            ],
    46	            "index": "pypi",
    47	            "version": "==0.7.2"
    48	        },
    49	        "protobuf": {
    50	            "hashes": [
    51	                "sha256:0bae429443cc4748be2aadfdaf9633297cfaeb24a9a02d0ab15849175ce90fab",
    52	                "sha256:24e3b6ad259544d717902777b33966a1a069208c885576254c112663e6a5bb0f",
    53	                "sha256:310a7aca6e7f257510d0c750364774034272538d51796ca31d42c3925d12a52a",
    54	                "sha256:52e586072612c1eec18e1174f8e3bb19d08f075fc2e3f91d3b16c919078469d0",
    55	                "sha256:73152776dc75f335c476d11d52ec6f0f6925774802cd48d6189f4d5d7fe753f4",
    56	                "sha256:7774bbbaac81d3ba86de646c39f154afc8156717972bf0450c9dbfa1dc8dbea2",
    57	                "sha256:82d7ac987715d8d1eb4068bf997f3053468e0ce0287e2729c30601feb6602fee",
    58	                "sha256:8eb9c93798b904f141d9de36a0ba9f9b73cc382869e67c9e642c0aba53b0fc07",
    59	                "sha256:adf0e4d57b33881d0c63bb11e7f9038f98ee0c3e334c221f0858f826e8fb0151",
    60	                "sha256:c40973a0aee65422d8cb4e7d7cbded95dfeee0199caab54d5ab25b63bce8135a",
    61	                "sha256:c77c974d1dadf246d789f6dad1c24426137c9091e930dbf50e0a29c1fcf00b1f",
    62	                "sha256:dd9aa4401c36785ea1b6fff0552c674bdd1b641319cb07ed1fe2392388e9b0d7",
    63	                "sha256:e11df1ac6905e81b815ab6fd518e79be0a58b5dc427a2cf7208980f30694b956",
    64	                "sha256:e2f8a75261c26b2f5f3442b0525d50fd79a71aeca04b5ec270fc123536188306",
    65	                "sha256:e512b7f3a4dd780f59f1bf22c302740e27b10b5c97e858a6061772668cd6f961",
    66	                "sha256:ef2c2e56aaf9ee914d3dccc3408d42661aaf7d9bb78eaa8f17b2e6282f214481",
    67	                "sha256:fac513a9dc2a74b99abd2e17109b53945e364649ca03d9f7a0b96aa8d1807d0a",
    68	                "sha256:fdfb6ad138dbbf92b5dbea3576d7c8ba7463173f7d2cb0ca1bd336ec88ddbd80"
    69	            ],
    70	            "index": "pypi",
    71	            "version": "==3.11.3"
    72	        },
    73	        "py4j": {
    74	            "hashes": [
    75	                "sha256:36ec57f43ff8ced260a18aa9a4e46c3500a730cac8860e259cbaa546c2b9db2f",
    76	                "sha256:859ba728a7bb43e9c2bf058832759fb97a598bb28cc12f34f5fc4abdec08ede6"
    77	            ],
    78	            "index": "pypi",
    79	            "version": "==0.10.9"
    80	        },
    81	        "six": {
    82	            "hashes": [
    83	                "sha256:236bdbdce46e6e6a3d61a337c0f8b763ca1e8717c03b369e87a7ec7ce1319c0a",
    84	                "sha256:8f3cd2e254d8f793e7f3d6d9df77b92252b52637291d0f0da013c76ea2724b6c"
    85	            ],
    86	            "index": "pypi",
    87	            "version": "==1.14.0"
    88	        },
    89	        "soupsieve": {
    90	            "hashes": [
    91	                "sha256:bdb0d917b03a1369ce964056fc195cfdff8819c40de04695a80bc813c3cfa1f5",
    92	                "sha256:e2c1c5dee4a1c36bcb790e0fabd5492d874b8ebd4617622c4f6a731701060dda"
    93	            ],
    94	            "version": "==1.9.5"
    95	        },
    96	        "tabulate": {
    97	            "hashes": [
    98	                "sha256:5470cc6687a091c7042cee89b2946d9235fe9f6d49c193a4ae2ac7bf386737c8"
    99	            ],
   100	            "index": "pypi",
   101	            "version": "==0.8.6"
   102	        }
   103	    },
   104	    "develop": {}
   105	}
==========================================================================

==========================================================================
File bazel-out/k8-fastbuild/bin/tools/dependencies/repo_deps_bzl_pipenv_interpreter.sh.runfiles/sync.log
--------------------------------------------------------------------------
     1	Creating a virtualenv for this project…
     2	Pipfile: /home/harora37/.cache/bazel/_bazel_harora37/4614b08a42facc1b3b190dce93d1ca57/execroot/com_ibm_monorepo/bazel-out/k8-fastbuild/bin/tools/dependencies/repo_deps_bzl_pipenv_interpreter.sh.runfiles/Pipfile
     3	Using /home/harora37/.cache/bazel/_bazel_harora37/4614b08a42facc1b3b190dce93d1ca57/execroot/com_ibm_monorepo/bazel-out/k8-fastbuild/bin/tools/python/build_python3.python (3.7.1) to create virtualenv…
⠙ Creating virtual environment...
     5 Successfully created virtual environment! 
     6	Virtualenv location: /home/harora37/.cache/bazel/_bazel_harora37/4614b08a42facc1b3b190dce93d1ca57/execroot/com_ibm_monorepo/bazel-out/k8-fastbuild/bin/tools/dependencies/repo_deps_bzl_pipenv_interpreter.sh.runfiles/.venv
     7	Installing dependencies from Pipfile.lock (684808)…
     8	Traceback (most recent call last):
     9	  File "/home/harora37/.cache/bazel/_bazel_harora37/4614b08a42facc1b3b190dce93d1ca57/execroot/com_ibm_monorepo/bazel-out/k8-fastbuild/bin/tools/python/build_python3.pipenv", line 10, in <module>
    10	    sys.exit(cli())
    11	  File "/home/harora37/.cache/bazel/_bazel_harora37/4614b08a42facc1b3b190dce93d1ca57/execroot/com_ibm_monorepo/bazel-out/k8-fastbuild/bin/tools/python/build_python3.interpreter/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 764, in __call__
    12	    return self.main(*args, **kwargs)
    13	  File "/home/harora37/.cache/bazel/_bazel_harora37/4614b08a42facc1b3b190dce93d1ca57/execroot/com_ibm_monorepo/bazel-out/k8-fastbuild/bin/tools/python/build_python3.interpreter/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 717, in main
    14	    rv = self.invoke(ctx)
    15	  File "/home/harora37/.cache/bazel/_bazel_harora37/4614b08a42facc1b3b190dce93d1ca57/execroot/com_ibm_monorepo/bazel-out/k8-fastbuild/bin/tools/python/build_python3.interpreter/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 1137, in invoke
    16	    return _process_result(sub_ctx.command.invoke(sub_ctx))
    17	  File "/home/harora37/.cache/bazel/_bazel_harora37/4614b08a42facc1b3b190dce93d1ca57/execroot/com_ibm_monorepo/bazel-out/k8-fastbuild/bin/tools/python/build_python3.interpreter/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 956, in invoke
    18	    return ctx.invoke(self.callback, **ctx.params)
    19	  File "/home/harora37/.cache/bazel/_bazel_harora37/4614b08a42facc1b3b190dce93d1ca57/execroot/com_ibm_monorepo/bazel-out/k8-fastbuild/bin/tools/python/build_python3.interpreter/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
    20	    return callback(*args, **kwargs)
    21	  File "/home/harora37/.cache/bazel/_bazel_harora37/4614b08a42facc1b3b190dce93d1ca57/execroot/com_ibm_monorepo/bazel-out/k8-fastbuild/bin/tools/python/build_python3.interpreter/lib/python3.7/site-packages/pipenv/vendor/click/decorators.py", line 64, in new_func
    22	    return ctx.invoke(f, obj, *args, **kwargs)
    23	  File "/home/harora37/.cache/bazel/_bazel_harora37/4614b08a42facc1b3b190dce93d1ca57/execroot/com_ibm_monorepo/bazel-out/k8-fastbuild/bin/tools/python/build_python3.interpreter/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
    24	    return callback(*args, **kwargs)
    25	  File "/home/harora37/.cache/bazel/_bazel_harora37/4614b08a42facc1b3b190dce93d1ca57/execroot/com_ibm_monorepo/bazel-out/k8-fastbuild/bin/tools/python/build_python3.interpreter/lib/python3.7/site-packages/pipenv/vendor/click/decorators.py", line 17, in new_func
    26	    return f(get_current_context(), *args, **kwargs)
    27	  File "/home/harora37/.cache/bazel/_bazel_harora37/4614b08a42facc1b3b190dce93d1ca57/execroot/com_ibm_monorepo/bazel-out/k8-fastbuild/bin/tools/python/build_python3.interpreter/lib/python3.7/site-packages/pipenv/cli/command.py", line 615, in sync
    28	    pypi_mirror=state.pypi_mirror,
    29	  File "/home/harora37/.cache/bazel/_bazel_harora37/4614b08a42facc1b3b190dce93d1ca57/execroot/com_ibm_monorepo/bazel-out/k8-fastbuild/bin/tools/python/build_python3.interpreter/lib/python3.7/site-packages/pipenv/core.py", line 2611, in do_sync
    30	    system=system,
    31	  File "/home/harora37/.cache/bazel/_bazel_harora37/4614b08a42facc1b3b190dce93d1ca57/execroot/com_ibm_monorepo/bazel-out/k8-fastbuild/bin/tools/python/build_python3.interpreter/lib/python3.7/site-packages/pipenv/core.py", line 1253, in do_init
    32	    pypi_mirror=pypi_mirror,
    33	  File "/home/harora37/.cache/bazel/_bazel_harora37/4614b08a42facc1b3b190dce93d1ca57/execroot/com_ibm_monorepo/bazel-out/k8-fastbuild/bin/tools/python/build_python3.interpreter/lib/python3.7/site-packages/pipenv/core.py", line 838, in do_install_dependencies
    34	    deps_list, procs, failed_deps_queue, requirements_dir, **install_kwargs
    35	  File "/home/harora37/.cache/bazel/_bazel_harora37/4614b08a42facc1b3b190dce93d1ca57/execroot/com_ibm_monorepo/bazel-out/k8-fastbuild/bin/tools/python/build_python3.interpreter/lib/python3.7/site-packages/pipenv/core.py", line 754, in batch_install
    36	    extra_indexes=extra_indexes
    37	  File "/home/harora37/.cache/bazel/_bazel_harora37/4614b08a42facc1b3b190dce93d1ca57/execroot/com_ibm_monorepo/bazel-out/k8-fastbuild/bin/tools/python/build_python3.interpreter/lib/python3.7/site-packages/pipenv/core.py", line 1421, in pip_install
    38	    c = delegator.run(pip_command, block=block, env=pip_config)
    39	  File "/home/harora37/.cache/bazel/_bazel_harora37/4614b08a42facc1b3b190dce93d1ca57/execroot/com_ibm_monorepo/bazel-out/k8-fastbuild/bin/tools/python/build_python3.interpreter/lib/python3.7/site-packages/pipenv/vendor/delegator.py", line 336, in run
    40	    c.run(block=block, binary=binary, cwd=cwd, env=env)
    41	  File "/home/harora37/.cache/bazel/_bazel_harora37/4614b08a42facc1b3b190dce93d1ca57/execroot/com_ibm_monorepo/bazel-out/k8-fastbuild/bin/tools/python/build_python3.interpreter/lib/python3.7/site-packages/pipenv/vendor/delegator.py", line 204, in run
    42	    s = PopenSpawn(self._popen_args, **pexpect_kwargs)
    43	  File "/home/harora37/.cache/bazel/_bazel_harora37/4614b08a42facc1b3b190dce93d1ca57/execroot/com_ibm_monorepo/bazel-out/k8-fastbuild/bin/tools/python/build_python3.interpreter/lib/python3.7/site-packages/pipenv/vendor/pexpect/popen_spawn.py", line 53, in __init__
    44	    self.proc = subprocess.Popen(cmd, **kwargs)
    45	  File "/home/harora37/.cache/bazel/_bazel_harora37/4614b08a42facc1b3b190dce93d1ca57/execroot/com_ibm_monorepo/bazel-out/k8-fastbuild/bin/tools/python/build_python3.interpreter/lib/python3.7/subprocess.py", line 769, in __init__
    46	    restore_signals, start_new_session)
    47	  File "/home/harora37/.cache/bazel/_bazel_harora37/4614b08a42facc1b3b190dce93d1ca57/execroot/com_ibm_monorepo/bazel-out/k8-fastbuild/bin/tools/python/build_python3.interpreter/lib/python3.7/subprocess.py", line 1516, in _execute_child
    48	    raise child_exception_type(errno_num, err_msg, err_filename)
    49	OSError: [Errno 8] Exec format error: '/home/harora37/.cache/bazel/_bazel_harora37/4614b08a42facc1b3b190dce93d1ca57/execroot/com_ibm_monorepo/bazel-out/k8-fastbuild/bin/tools/dependencies/repo_deps_bzl_pipenv_interpreter.sh.runfiles/.venv/bin/pip'
==========================================================================
Installing dependencies from Pipfile.lock (684808)…
Target //tools/dependencies:repo_deps failed to build
Use --verbose_failures to see the command lines of failed build steps.

[gaborbernat]

Seems pipenv is obfuscating the virtualenv creation so there's not enough information in the output to tell for sure (ideally we'd want the -vv output of the virtualenv creation). Can you get the virtual environment creation logs? I'd suggest also trying setting the VIRTUALENV_SEEDER=pip env-var and see if that also triggers this behaviour. Thanks!

[systemlogic]

@gaborbernat Can I request you to check with your team that get-pip.py which is updated on 10th feb is compiled on Ubuntu not on mac. OSError: [Errno 8] Exec format error generally comes when python binary compiled on one OS and try to run on incompatible OS.
Or if there is a reference to older get-pip.py is there .

[gaborbernat]

virtualenv does not use get-pip.py at all, so it's not related. I'd still want to see those creation logs to determine the underlying issue here.

[systemlogic]

@gaborbernat can you send me set of command ?
I am asking since our CI server is ruuning on ubuntu image in Jenkins and we don't have access to download that environment.

[gaborbernat]

Did you try setting the environment variable VIRTUALENV_SEEDER to pip?

[systemlogic]

NO, I have to prepare a script to run on black box, which I don't have access. I can only create PR to debug the issue. No local

[systemlogic]

Actually we are using pipenv to create a grap and install the packages.

[gaborbernat]

In that PR you can just set that env-var, not?

[systemlogic]

@gaborbernat any idea if some value is changed for shebang length. String is getting truncated after 80, where it was allowing long length

[gaborbernat]

Yeah, that regressed sadly, we no longer support long paths; will need a bugfix for that, see #997 .

[systemlogic]

@gaborbernat can we have long path restored or at least a way to override default value?

[gaborbernat]

As soon as I get to put in a PR; sadly there's only one of me and quite a few such issues... but working on it.

[gaborbernat]

#1579 this should do it.

[systemlogic]

thanks, when binary will be updated ?

[gaborbernat]

#1580 work in progress

[gaborbernat]

Hello, the fix for the long path has been released via virtualenv 20.0.2; see https://pypi.org/project/virtualenv/20.0.2 . Please give a try and report back if your issue has not been addressed; if not, please comment here. We want to apologize for the inconvenience this has caused you and say thanks for having patience while we resolve the unexpected bugs with this new major release.

[systemlogic]

Perfectly fine. Our CI is restored.