sys.executable might be Vim itself with embedded Python (UnpicklingError)

[tonitch]

(sorry for miss spellings, I'm french and I'll try my best to explain my issue)

Issue

When I try to or when jedi-vim want to help me, a new window of gvim pop-up with main.py oppened from jedi's file instead of the list of propositions and sometimes, when I go back to the previews window, gvim stop working...

Steps to reproduce

here is my _vimrc ::

source $VIMRUNTIME/vimrc_example.vim

let python_highlight_all=1
syntax on

"Enable folding + space to fold
set foldmethod=indent
set foldlevel=99
nnoremap <space> za

set diffexpr=MyDiff()
function MyDiff()
  let opt = '-a --binary '
  if &diffopt =~ 'icase' | let opt = opt . '-i ' | endif
  if &diffopt =~ 'iwhite' | let opt = opt . '-b ' | endif
  let arg1 = v:fname_in
  if arg1 =~ ' ' | let arg1 = '"' . arg1 . '"' | endif
  let arg1 = substitute(arg1, '!', '\!', 'g')
  let arg2 = v:fname_new
  if arg2 =~ ' ' | let arg2 = '"' . arg2 . '"' | endif
  let arg2 = substitute(arg2, '!', '\!', 'g')
  let arg3 = v:fname_out
  if arg3 =~ ' ' | let arg3 = '"' . arg3 . '"' | endif
  let arg3 = substitute(arg3, '!', '\!', 'g')
  if $VIMRUNTIME =~ ' '
    if &sh =~ '\<cmd'
      if empty(&shellxquote)
        let l:shxq_sav = ''
        set shellxquote&
      endif
      let cmd = '"' . $VIMRUNTIME . '\diff"'
    else
      let cmd = substitute($VIMRUNTIME, ' ', '" ', '') . '\diff"'
    endif
  else
    let cmd = $VIMRUNTIME . '\diff'
  endif
  let cmd = substitute(cmd, '!', '\!', 'g')
  silent execute '!' . cmd . ' ' . opt . arg1 . ' ' . arg2 . ' > ' . arg3
  if exists('l:shxq_sav')
    let &shellxquote=l:shxq_sav
  endif
endfunction

set nu
set guifont=Courier_new:h11
set encoding=utf-8
set nocompatible
color desert

filetype off
set rtp+=%USERPROFILE%/vim/bundle/Vundle.vim/
call vundle#begin('%USERPROFILE%/vim/bundle/')

Plugin 'vundleVim/Vundle.vim'
Plugin 'scrooloose/nerdtree'
Plugin 'itchyny/lightline.vim'
Plugin 'mattn/emmet-vim'
Plugin 'Xuyuanp/nerdtree-git-plugin'
Plugin 'altercation/vim-colors-solarized'
Plugin 'jiangmiao/auto-pairs'
Plugin 'kien/rainbow_parentheses.vim'
Plugin 'vim-syntastic/syntastic'
Plugin 'nvie/vim-flake8'
Plugin 'tpope/vim-dispatch'
Plugin 'https://github.com/StanAngeloff/php.vim.git'
Plugin 'joonty/vim-phpqa'
Plugin 'davidhalter/jedi-vim'

call vundle#end()
filetype plugin indent on

"if has('gui_running')
"	set background=dark
"	colorscheme solarized
"else
"	colorscheme zenburn
"endif

filetype plugin on
set omnifunc=syntaxcomplete#Complete

Output of “:verbose JediDebugInfo”

when I do this command, the same windows apear with

import sys
import os


def _get_paths():
    # Get the path to jedi.
    _d = os.path.dirname
    _jedi_path = _d(_d(_d(_d(_d(__file__)))))
    _parso_path = sys.argv[1]
    # The paths are the directory that jedi and parso lie in.
    return {'jedi': _jedi_path, 'parso': _parso_path}


# Remove the first entry, because it's simply a directory entry that equals
# this directory.
del sys.path[0]

if sys.version_info > (3, 4):
    from importlib.machinery import PathFinder

    class _ExactImporter(object):
        def __init__(self, path_dct):
            self._path_dct = path_dct

        def find_module(self, fullname, path=None):
            if path is None and fullname in self._path_dct:
                p = self._path_dct[fullname]
                loader = PathFinder.find_module(fullname, path=[p])
                return loader
            return None

    # Try to import jedi/parso.
    sys.meta_path.insert(0, _ExactImporter(_get_paths()))
    from jedi.evaluate.compiled import subprocess  # NOQA
    sys.meta_path.pop(0)
else:
    import imp

    def load(name):
        paths = list(_get_paths().values())
        fp, pathname, description = imp.find_module(name, paths)
        return imp.load_module(name, fp, pathname, description)

    load('parso')
    load('jedi')
    from jedi.evaluate.compiled import subprocess  # NOQA

# Retrieve the pickle protocol.
pickle_protocol = int(sys.argv[2])
# And finally start the client.
subprocess.Listener(pickle_protocol).listen()

[davidhalter]

I have no idea what this could be. lol. This sounds like such a weird issue. Anyone else?

[lboulard]

+1

Reverting to 0.9.0 from master fixed the issue while editing Python 3 code.
Happen also with function parameters completion.

This is GVim on Windows (all 64bit, GVim and Python) and a new Gvim instance opens on jedi\jedi\evaluate\compiled\subprocess\__main__.py from submodule jedi.

[davidhalter]

@micbou Do you have any idea what this could be? I think this might be related to how Jedi starts subprocesses, but I have no idea about these Windows issues. Contrary to YCM Jedi is run directly in VIM's embedded Python.

[lboulard]

I can no longer reproduce issue editing same python source file after a JediClearCache and a fresh new GVim instance.

[micbou]

I can reproduce the issue by running the :JediDebugInfo command. This happens because sys.executable inside Vim returns the path to the Vim executable on Windows so, instead of starting a Python environment, Jedi is starting a new instance of Vim on that __main__.py file. A solution would be to overwrite sys.executable with the right path on Windows:

sys.executable = os.path.join(sys.prefix, 'python.exe')

[davidhalter]

We already do this for prefix anyway, because it gets messed up by virtualenvs, so this would be a good solution I think.

[yexianingyue]

hello .how did you solve it ? I had the same problem.

[AndreyZhdanov]

put
py3 import os; sys.executable=os.path.join(sys.prefix, 'python.exe')

into .vimrc or _vimrc file

[davidhalter]

@AndreyZhdanov Could you try to put that to the beginning of pythonx/jedi_vim.py? And see if it works?

[AndreyZhdanov]

@davidhalter Yes. I put
sys.executable=os.path.join(sys.prefix, 'python.exe')

after import os – line 9 and it works fine for me.

But I still prefer to have this piece in _vimrc beer in mind update of the script. And it looks like vim bug not issue with jedi-vim.

What is your opinion ?

[blueyed]

1. Is this broken/weird already with :py import sys; print(sys.executable) and :py3 import sys; print(sys.executable)?

2. Does it only happen with gvim, or also vim on the terminal/cmd.exe (if you have that)?

3. how is Python built into Vim? Dynamically? (look for "py" in :version)

Reverting to 0.9.0 from master fixed the issue while editing Python 3 code.

It would be useful to have this git-bisected to the commi where it started to happen.

[AndreyZhdanov]

1. I don't understand 1st question, the script is loading during startup, command in 1st point could be executed after. When I put these commands in _vimrc everything is fine.
2. With gvim only.
3. Dynamically.

[blueyed]

@AndreyZhdanov
Re 1.: I've meant to execute the command in a running gvim instance

[tngreene]

Can confirm: Reverting to 0.9.0 fixed the problem

[blueyed]

@tngreene
Can you bisect this to a particular commit?

[tngreene]

4c430ed536b6484f7c63929a4426d64f981d994e is the first bad commit
commit 4c430ed536b6484f7c63929a4426d64f981d994e
Author: Daniel Hahler <github@thequod.de>
Date:   Sat Jul 28 01:46:12 2018 +0200

    Improve JediDebugInfo for envs (#858)

    Improve JediDebugInfo for envs

    This is taken out of https://github.com/davidhalter/jedi-vim/pull/836.

Fortunately, this does not scream implausible to me.

[blueyed]

@tngreene
Does it happen (only) with :JediDebugInfo for you?
I assume 4c430ed triggers it for this case, since it uses jedi.api.environment.

Can you post the output from running the Vim commands from the first point in #870 (comment), please?

[micbou]

@blueyed Did you miss #870 (comment)? This comment explains the issue and gives a solution so it should only be a matter of implementing it.

[tngreene]

@tngreene
Does it happen (only) with :JediDebugInfo for you?
I assume 4c430ed triggers it for this case, since it uses jedi.api.environment.

Can you post the output from running the Vim commands from the first point in #870 (comment), please?

Running with :py breaks, because I don't have Python 2 installed. with :py3 it prints C:\Vim\gvim.exe

[blueyed]

@micbou
I wanted to understand the issue first.

It is clear now: since Python is linked into GVim gvim.exe appears to be the executable.
Sounds a bit like a GVim bug / build issue though: on Linux it returns a python binary with gvim and vim (dynamically linked also). Please consider reporting it with Vim also.

As for fixing this in jedi/jedi-vim: please submit a PR.

[micbou]

Sounds a bit like a GVim bug / build issue though: on Linux it returns a python binary with gvim and vim (dynamically linked also). Please consider reporting it with Vim also.

This has already been discussed on the Vim mailing list and it's not an issue with Vim but a peculiarity of Python on Windows: when a program is embedding Python, sys.executable points to the executable of that program instead of the interpreter.

As for fixing this in jedi/jedi-vim: please submit a PR.

I would if I were a user of jedi-vim but I am not.

[tngreene]

put
py3 import os; sys.executable=os.path.join(sys.prefix, 'python.exe')

into .vimrc or _vimrc file

You missed importing sys

py3 import os; import sys; sys.executable=os.path.join(sys.prefix, 'python.exe')

This made it work for me. Thank you! Looking forward to an official fix, however! That way no one else gets confused.

[lboulard]

Hi,

Finally, I have again issue of launching gvim.exe instead of python.exe.
I have been able to bisect commit of issue in davidhalter/jedi@f9cbc65 (from pythonx/jedi submodule).

Reverting this commit from version v0.13.1 of jedi allow JediDebugInfo to work correctly.

Hope this help.

[blueyed]

@lboulard
Thanks for finding this.

@davidhalter
You said:

We already do this for prefix anyway, because it gets messed up by virtualenvs, so this would be a good solution I think.

Where is that?

I think a fix needs to be applied to https://github.com/davidhalter/jedi/blob/3bdb941daa2d87947fa265ebcf7d94f68efbc42d/jedi/api/environment.py#L133 (if it should be done for Jedi, which makes sense, since Python might be embedded also in non-Vim apps).

[j5shi]

Any update about the fix?

[gerroon]

I have the same issue.

[davidhalter]

Where is that?

@blueyed https://github.com/davidhalter/jedi-vim/blob/master/pythonx/jedi_vim.py#L31

I think a fix needs to be applied to https://github.com/davidhalter/jedi/blob/3bdb941daa2d87947fa265ebcf7d94f68efbc42d/jedi/api/environment.py#L133 (if it should be done for Jedi, which makes sense, since Python might be embedded also in non-Vim apps).

I tend to agree. But what would be a good fix? I'm not really sure how to do get the executable path of Python if sys.executable was overridden.

[blueyed]

@davidhalter
LGTM. But s/parso/jedi in davidhalter/jedi#1063 (comment)

[scrosland]

@scrosland Thanks for that information! I have used your information in the latest commits to improve the environment selection. Could you please test the latest Jedi commits and see if it still happens?

I updated vim-jedi (and jedi) and it works correctly now without my previous vimrc workaround. Thanks!