activate_this.py failed for python2 virtualenvs

[NyaMisty]

Issue

It seems recently pipenv introduced a new type of activate_this.py. On windows the content of activate_this.py has something like this:

prev_length = len(sys.path)
for lib in "'..\\Lib\\site-packages".split(os.pathsep):
    path = os.path.realpath(os.path.join(bin_dir, lib))
    site.addsitedir(path.decode("utf-8") if "'yes" else path)
sys.path[:] = sys.path[prev_length:] + sys.path[0:prev_length]

As you can see the "'..\Lib\site-packages" is obviously wrong.

Environment

Provide at least:

• OS: Windows 10

• pip list of the host python where virtualenv is installed:

  virtualenv          20.0.18
  virtualenv-clone    0.5.4

Output of the virtual environment creation

As I'm using virtualenv through pipenv, so I failed to grab the virtualenv output

[  ==] Creating virtual environment...created virtual environment CPython2.7.17.final.0-64 in 641ms

  creator CPython2Windows(dest=C:\Users\win10\.virtualenvs\win10-obmjl69F, clear=False, global=False)

  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=C:\Users\win10\AppData\Local\pypa\virtualenv\seed-app-data\v1.0.1)

  activators BashActivator,BatchActivator,FishActivator,PowerShellActivator,PythonActivator

However I've located the related code and wrote out its runtime variable information:
The following is the output of _repr_unicode function in src/virtualenv/activation/python/__init__.py

'(win10) 
'C:\\Users\\win10\\.virtualenvs\\win10-obmjl69F
'win10-obmjl69F
'Scripts
';
'..\\Lib\\site-packages
'yes

As you can see, there's an additional ' before each item. I've done a small experiment on python 3.6 and 3.7:

>>> value = "..\\123456"
>>> repr(value.encode("utf-8"))
"b'..\\\\123456'"
>>> repr(value.encode("utf-8"))[1:-1]
"'..\\\\123456"
>>>

I believe there's something wrong with this function. This function is introduced in PR #1503

[gaborbernat]

virtualenv-clone    0.5.4

virtualenv 20 no longer supports the relocatable feature. The fact that that package hasn't been updated in a while feels to me like it's no longer compatible with virtualenv 20. Please uninstall and try without.

As I'm using virtualenv through pipenv, so I failed to grab the virtualenv output

You can still set environment variables to switch to verbose output per https://virtualenv.pypa.io/en/latest/cli_interface.html#environment-variables

It seems recently pipenv introduced a new type of activate_this.py.

Link to this feature, description of this feature?

Can you replicate this without pipenv by just doing virtualenv call? Provide the python version of both the host and target python.

[NyaMisty]

1. There's obviously nothing to do with the virtualenv-clone, as this is activator-related.
2. I've switched to the bare virtualenv call, and here's output. But that won't help at all, as the python activator has no output

109 setup logging to NOTSET [DEBUG report:43]
109 find interpreter for spec PythonSpec(major=2, minor=7) [INFO builtin:44]
109 proposed PythonInfo(spec=CPython3.6.8.final.0-64, exe=c:\users\win10\appdata\local\programs\python\python36\python.exe, platform=win32, version='3.6.8 (tags/v3.6.8:3c6b436a57, Dec 24 2018, 00:16:47) [MSC v.1916 64 bit (AMD64)]', encoding_fs_io=utf-8-utf-8) [INFO builtin:50]
125 Attempting to acquire lock 1774255880736 on C:\Users\win10\AppData\Local\pypa\virtualenv\py_info\20.0.18\5c3b02fa24688d32a1d09351b7f8eac82a87272ad081601c73caead00a8e1fc4.lock [DEBUG filelock:270]
125 Lock 1774255880736 acquired on C:\Users\win10\AppData\Local\pypa\virtualenv\py_info\20.0.18\5c3b02fa24688d32a1d09351b7f8eac82a87272ad081601c73caead00a8e1fc4.lock [INFO filelock:274]
125 get PythonInfo from C:\Users\win10\AppData\Local\pypa\virtualenv\py_info\20.0.18\5c3b02fa24688d32a1d09351b7f8eac82a87272ad081601c73caead00a8e1fc4.json for C:\Python27\python.exe [DEBUG cached_py_info:79]
125 Attempting to release lock 1774255880736 on C:\Users\win10\AppData\Local\pypa\virtualenv\py_info\20.0.18\5c3b02fa24688d32a1d09351b7f8eac82a87272ad081601c73caead00a8e1fc4.lock [DEBUG filelock:315]
125 Lock 1774255880736 released on C:\Users\win10\AppData\Local\pypa\virtualenv\py_info\20.0.18\5c3b02fa24688d32a1d09351b7f8eac82a87272ad081601c73caead00a8e1fc4.lock [INFO filelock:318]
125 proposed Pep514PythonInfo(spec=CPython2.7.17.final.0-64, exe=C:\Python27\python.exe, platform=win32, version='2.7.17 (v2.7.17:c2f86d86e6, Oct 19 2019, 21:01:17) [MSC v.1500 64 bit (AMD64)]', encoding_fs_io=mbcs-None) [INFO builtin:50]
125 accepted Pep514PythonInfo(spec=CPython2.7.17.final.0-64, exe=C:\Python27\python.exe, platform=win32, version='2.7.17 (v2.7.17:c2f86d86e6, Oct 19 2019, 21:01:17) [MSC v.1500 64 bit (AMD64)]', encoding_fs_io=mbcs-None) [DEBUG builtin:52]
125 symlink on filesystem does work [DEBUG info:47]
125 filesystem is not case-sensitive [DEBUG info:28]
234 create virtual environment via CPython2Windows(dest=C:\Users\win10\test, clear=False, global=False) [INFO session:52]
234 create folder C:\Users\win10\test\Lib\site-packages [DEBUG _sync:21]
234 create folder C:\Users\win10\test\Scripts [DEBUG _sync:21]
234 write C:\Users\win10\test\pyvenv.cfg [DEBUG pyenv_cfg:34]
234     home = C:\Python27 [DEBUG pyenv_cfg:38]
234     implementation = CPython [DEBUG pyenv_cfg:38]
234     version_info = 2.7.17.final.0 [DEBUG pyenv_cfg:38]
234     virtualenv = 20.0.18 [DEBUG pyenv_cfg:38]
234     include-system-site-packages = false [DEBUG pyenv_cfg:38]
234     base-prefix = C:\Python27 [DEBUG pyenv_cfg:38]
234     base-exec-prefix = C:\Python27 [DEBUG pyenv_cfg:38]
234     base-executable = C:\Python27\python.exe [DEBUG pyenv_cfg:38]
234 copy C:\Python27\python.exe to C:\Users\win10\test\Scripts\python.exe [DEBUG _sync:48]
234 copy C:\Python27\pythonw.exe to C:\Users\win10\test\Scripts\pythonw.exe [DEBUG _sync:48]
234 symlink C:\Python27\Lib\os.py to C:\Users\win10\test\Lib\os.py [DEBUG _sync:40]
249 symlink C:\Python27\Lib\os.pyc to C:\Users\win10\test\Lib\os.pyc [DEBUG _sync:40]
249 symlink directory C:\Python27\Include to C:\Users\win10\test\Include [DEBUG _sync:40]
249 symlink directory C:\Python27\libs to C:\Users\win10\test\libs [DEBUG _sync:40]
249 create virtualenv import hook file C:\Users\win10\test\Lib\site-packages\_virtualenv.pth [DEBUG api:78]
249 create C:\Users\win10\test\Lib\site-packages\_virtualenv.py [DEBUG api:81]
249 ============================== target debug ============================== [DEBUG session:54]
249 debug via 'C:\Users\win10\test\Scripts\python.exe' 'c:\users\win10\appdata\local\programs\python\python36\lib\site-packages\virtualenv\create\debug.py' [DEBUG creator:187]
249 {
  "sys": {
    "executable": "C:\\Users\\win10\\test\\Scripts\\python.exe",
    "_base_executable": null,
    "prefix": "C:\\Users\\win10\\test",
    "base_prefix": "C:\\Python27",
    "real_prefix": "C:\\Python27",
    "exec_prefix": "C:\\Users\\win10\\test",
    "base_exec_prefix": "C:\\Python27",
    "path": [
      "C:\\Windows\\SYSTEM32\\python27.zip",
      "C:\\Python27\\DLLs",
      "C:\\Python27\\lib",
      "C:\\Python27\\lib\\plat-win",
      "C:\\Python27\\lib\\lib-tk",
      "C:\\Users\\win10\\test\\Scripts",
      "C:\\Users\\win10\\test",
      "C:\\Users\\win10\\test\\lib\\site-packages"
    ],
    "meta_path": [
      "<class '_virtualenv._VirtualenvImporter'>"
    ],
    "fs_encoding": "mbcs",
    "io_encoding": null
  },
  "version": "2.7.17 (v2.7.17:c2f86d86e6, Oct 19 2019, 21:01:17) [MSC v.1500 64 bit (AMD64)]",
  "os": "<module 'os' from 'C:\\Python27\\lib\\os.pyc'>",
  "site": "<module 'site' from 'C:\\Python27\\lib\\site.pyc'>",
  "datetime": "<module 'datetime' (built-in)>",
  "math": "<module 'math' (built-in)>",
  "json": "<module 'json' from 'C:\\Python27\\lib\\json\\__init__.pyc'>"
} [DEBUG session:55]
329 add seed packages via FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=C:\Users\win10\AppData\Local\pypa\virtualenv\seed-app-data\v1.0.1) [INFO session:59]
329 Attempting to acquire lock 1774263916696 on C:\Users\win10\AppData\Local\pypa\virtualenv\seed-app-data\v1.0.1\2.7\wheels.lock [DEBUG filelock:270]
329 Lock 1774263916696 acquired on C:\Users\win10\AppData\Local\pypa\virtualenv\seed-app-data\v1.0.1\2.7\wheels.lock [INFO filelock:274]
344 get bundled wheel c:\users\win10\appdata\local\programs\python\python36\lib\site-packages\virtualenv\seed\embed\wheels\pip-20.0.2-py2.py3-none-any.whl [DEBUG acquire:51]
344 get bundled wheel c:\users\win10\appdata\local\programs\python\python36\lib\site-packages\virtualenv\seed\embed\wheels\setuptools-44.1.0-py2.py3-none-any.whl [DEBUG acquire:51]
344 get bundled wheel c:\users\win10\appdata\local\programs\python\python36\lib\site-packages\virtualenv\seed\embed\wheels\wheel-0.34.2-py2.py3-none-any.whl [DEBUG acquire:51]
344 install wheel from wheel C:\Users\win10\AppData\Local\pypa\virtualenv\seed-app-data\v1.0.1\2.7\wheels\wheel-0.34.2-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:46]
344 copy directory C:\Users\win10\AppData\Local\pypa\virtualenv\seed-app-data\v1.0.1\2.7\image\CopyPipInstall\wheel-0.34.2-py2.py3-none-any\wheel to C:\Users\win10\test\Lib\site-packages\wheel [DEBUG _sync:48]
344 install setuptools from wheel C:\Users\win10\AppData\Local\pypa\virtualenv\seed-app-data\v1.0.1\2.7\wheels\setuptools-44.1.0-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:46]
344 install pip from wheel C:\Users\win10\AppData\Local\pypa\virtualenv\seed-app-data\v1.0.1\2.7\wheels\pip-20.0.2-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:46]
344 copy C:\Users\win10\AppData\Local\pypa\virtualenv\seed-app-data\v1.0.1\2.7\image\CopyPipInstall\setuptools-44.1.0-py2.py3-none-any\easy_install.py to C:\Users\win10\test\Lib\site-packages\easy_install.py [DEBUG _sync:48]
344 copy directory C:\Users\win10\AppData\Local\pypa\virtualenv\seed-app-data\v1.0.1\2.7\image\CopyPipInstall\pip-20.0.2-py2.py3-none-any\pip to C:\Users\win10\test\Lib\site-packages\pip [DEBUG _sync:48]
344 copy directory C:\Users\win10\AppData\Local\pypa\virtualenv\seed-app-data\v1.0.1\2.7\image\CopyPipInstall\setuptools-44.1.0-py2.py3-none-any\pkg_resources to C:\Users\win10\test\Lib\site-packages\pkg_resources [DEBUG _sync:48]
365 copy directory C:\Users\win10\AppData\Local\pypa\virtualenv\seed-app-data\v1.0.1\2.7\image\CopyPipInstall\wheel-0.34.2-py2.py3-none-any\wheel-0.34.2.dist-info to C:\Users\win10\test\Lib\site-packages\wheel-0.34.2.dist-info [DEBUG _sync:48]
380 copy directory C:\Users\win10\AppData\Local\pypa\virtualenv\seed-app-data\v1.0.1\2.7\image\CopyPipInstall\setuptools-44.1.0-py2.py3-none-any\setuptools to C:\Users\win10\test\Lib\site-packages\setuptools [DEBUG _sync:48]
380 copy C:\Users\win10\AppData\Local\pypa\virtualenv\seed-app-data\v1.0.1\2.7\image\CopyPipInstall\wheel-0.34.2-py2.py3-none-any\wheel-0.34.2.virtualenv to C:\Users\win10\test\Lib\site-packages\wheel-0.34.2.virtualenv [DEBUG _sync:48]
380 generated console scripts wheel2.exe wheel.exe wheel-2.7.exe [DEBUG base:53]
474 copy directory C:\Users\win10\AppData\Local\pypa\virtualenv\seed-app-data\v1.0.1\2.7\image\CopyPipInstall\setuptools-44.1.0-py2.py3-none-any\setuptools-44.1.0.dist-info to C:\Users\win10\test\Lib\site-packages\setuptools-44.1.0.dist-info [DEBUG _sync:48]
474 copy C:\Users\win10\AppData\Local\pypa\virtualenv\seed-app-data\v1.0.1\2.7\image\CopyPipInstall\setuptools-44.1.0-py2.py3-none-any\setuptools-44.1.0.virtualenv to C:\Users\win10\test\Lib\site-packages\setuptools-44.1.0.virtualenv [DEBUG _sync:48]
490 generated console scripts easy_install2.exe easy_install.exe easy_install-2.7.exe [DEBUG base:53]
708 copy directory C:\Users\win10\AppData\Local\pypa\virtualenv\seed-app-data\v1.0.1\2.7\image\CopyPipInstall\pip-20.0.2-py2.py3-none-any\pip-20.0.2.dist-info to C:\Users\win10\test\Lib\site-packages\pip-20.0.2.dist-info [DEBUG _sync:48]
708 copy C:\Users\win10\AppData\Local\pypa\virtualenv\seed-app-data\v1.0.1\2.7\image\CopyPipInstall\pip-20.0.2-py2.py3-none-any\pip-20.0.2.virtualenv to C:\Users\win10\test\Lib\site-packages\pip-20.0.2.virtualenv [DEBUG _sync:48]
724 generated console scripts pip.exe pip2.7.exe pip-2.7.exe pip2.exe [DEBUG base:53]
724 Attempting to release lock 1774263916696 on C:\Users\win10\AppData\Local\pypa\virtualenv\seed-app-data\v1.0.1\2.7\wheels.lock [DEBUG filelock:315]
724 Lock 1774263916696 released on C:\Users\win10\AppData\Local\pypa\virtualenv\seed-app-data\v1.0.1\2.7\wheels.lock [INFO filelock:318]
724 add activators for Bash, Batch, Fish, PowerShell, Python [INFO session:65]
740 write C:\Users\win10\test\pyvenv.cfg [DEBUG pyenv_cfg:34]
740     home = C:\Python27 [DEBUG pyenv_cfg:38]
740     implementation = CPython [DEBUG pyenv_cfg:38]
740     version_info = 2.7.17.final.0 [DEBUG pyenv_cfg:38]
740     virtualenv = 20.0.18 [DEBUG pyenv_cfg:38]
740     include-system-site-packages = false [DEBUG pyenv_cfg:38]
740     base-prefix = C:\Python27 [DEBUG pyenv_cfg:38]
740     base-exec-prefix = C:\Python27 [DEBUG pyenv_cfg:38]
740     base-executable = C:\Python27\python.exe [DEBUG pyenv_cfg:38]
740 created virtual environment CPython2.7.17.final.0-64 in 646ms
  creator CPython2Windows(dest=C:\Users\win10\test, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=C:\Users\win10\AppData\Local\pypa\virtualenv\seed-app-data\v1.0.1)
  activators BashActivator,BatchActivator,FishActivator,PowerShellActivator,PythonActivator [WARNING __main__:21]

3. Saying "new type" I'm refering to the layout of activate_this.py. You rewrote the PythonGenerator in your PR, changes the activate_this.py from this one (works flawlessly): (from virtualenv 15.X)

"""By using execfile(this_file, dict(__file__=this_file)) you will
activate this virtualenv environment.

This can be used when you must use an existing Python interpreter, not
the virtualenv bin/python
"""

try:
    __file__
except NameError:
    raise AssertionError(
        "You must run this like execfile('path/to/activate_this.py', dict(__file__='path/to/activate_this.py'))")
import sys
import os

old_os_path = os.environ.get('PATH', '')
os.environ['PATH'] = os.path.dirname(os.path.abspath(__file__)) + os.pathsep + old_os_path
base = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
if sys.platform == 'win32':
    site_packages = os.path.join(base, 'Lib', 'site-packages')
else:
    site_packages = os.path.join(base, 'lib', 'python%s' % sys.version[:3], 'site-packages')
prev_sys_path = list(sys.path)
import site
site.addsitedir(site_packages)
sys.real_prefix = sys.prefix
sys.prefix = base
# Move the added items to the front of the path:
new_sys_path = []
for item in list(sys.path):
    if item not in prev_sys_path:
        new_sys_path.append(item)
        sys.path.remove(item)
sys.path[:0] = new_sys_path

to this one

# -*- coding: utf-8 -*-
"""Activate virtualenv for current interpreter:

Use exec(open(this_file).read(), {'__file__': this_file}).

This can be used when you must use an existing Python interpreter, not the virtualenv bin/python.
"""
import os
import site
import sys

try:
    abs_file = os.path.abspath(__file__)
except NameError:
    raise AssertionError("You must use exec(open(this_file).read(), {'__file__': this_file}))")

bin_dir = os.path.dirname(abs_file)
base = bin_dir[: -len("'Scripts") - 1]  # strip away the bin part from the __file__, plus the path separator

# prepend bin to PATH (this file is inside the bin directory)
os.environ["PATH"] = os.pathsep.join([bin_dir] + os.environ.get("PATH", "").split(os.pathsep))
os.environ["VIRTUAL_ENV"] = base  # virtual env is right above bin directory

# add the virtual environments libraries to the host python import mechanism
prev_length = len(sys.path)
for lib in "'..\\Lib\\site-packages".split(os.pathsep):
    path = os.path.realpath(os.path.join(bin_dir, lib))
    site.addsitedir(path.decode("utf-8") if "'yes" else path)
sys.path[:] = sys.path[prev_length:] + sys.path[0:prev_length]

sys.real_prefix = sys.prefix
sys.prefix = base

[NyaMisty]

The host python is Python 3.6.8, also tested on 3.7.2. The target python is already shown in the output (2.7.17)
That "'" can only be generated from PythonGenerator, and that generator receives no parameter that can affect its output like this. I've also gave you a simple example why your code doesn't work. I think it's obvious that the bug locates there :)

[gaborbernat]

1. You rewrote the PythonGenerator in your PR, changes the activate_this.py from this one (works flawlessly):

I consider this kind of aggressive finger-pointing non-constructive here. I've simply asked for some more information, not to be told of; sigh you're welcome for free labour.

[NyaMisty]

Well I'm not trying to finger-pointing you. I'm not familiar with English, and it's my fault to let you feeling so :( Please pardon me for my bad English.
What I wanted to mean is the change is happed in your PR, so instead of trying to be rude here, I'm suggesting you go over your code again, so it would be easier to understand (it's always easier to understand our own code, right? :) )

[gaborbernat]

Hello, this now has been released via https://pypi.org/project/virtualenv/20.0.19/