Problem running debugpy in embedded python interpreter

[nicelifeBS]

Environment data

• debugpy version: 1.0.0b9
• OS and version: Windows 10
• Python version (& distribution if applicable, e.g. Anaconda): Blender 2.82, Nuke 11, Maya2018
• Using VS Code or Visual Studio: 1.45.0

Actual behavior

Remote debugging works running it in a python environment in a shell. But when run through an application with embedded python interpreter it starts a new instance of this application and it cannot connect.

I think the problem is that sys.executable is not returning the path to the python interpreter but the path of the application executable itself. And because of this the arguments passed with subprocess are not recognized.

debugpy/src/debugpy/server/api.py

Line 103 in 7a2891b

sys.executable,

If I replace sys.executable with the path to the embedded interpreter remote debugging works as expected.

An older version of ptvsd worked out of the box btw: microsoft/ptvsd@0825dbe

[tanant]

dumb question - do we need the specific sys.executable that the process/app running is using, or for the purposes of this, do we just need a valid python interpreter to mediate the comms back and forth?

[karthiknadig]

@tanant In the case mentioned above it is needed for comms. For the above case, attach using process id would probably work. Essentially start debug adapter in server mode.

python  /path/to/debugpy/adapter --host 0.0.0.0 --port 5678

Then from VSCode, use the following to connect:

        {
            "name": "Remote Attach",
            "type": "python",
            "request": "attach",
            "debugServer": 5678,
            "processId": "", // <---- put process id of the process that has the embedded python
            "pathMappings": [{
                "localRoot": "${workspaceFolder}",
                "remoteRoot": "."
            }]
        }

This way you will use the python executable for comms with VS Code, and it should be able to load the debugger where the embedded python. This uses a different pathway where we don't try to start the python executable.

[puremourning]

I think the problem is that sys.executable is not returning the path to the python interpreter but the path of the application executable itself. And because of this the arguments passed with subprocess are not recognized.

I found the same thing. This was my solution. It's absolutely not art: https://github.com/puremourning/vimspector/blob/master/python3/vimspector/developer.py#L23

[fabioz]

Well, I think that as @karthiknadig mentioned, the approach currently supported in this case is the attach by pid.

I think that we could extend that to accept a python_exe in debugpy.listen (which would default to sys.executable) so that the user could customize it without fiddling with sys.executable as @puremourning mentioned.

One thing to note though is that we don't support automatically attaching to subprocesess in this case (we check if the executable basename has python on it so that we don't inadvertently attach to some non-python process such as calc and crash it).

[int19h]

This kind of stuff should go into debugpy.configure() if we do it.

What's not clear to me is why the adapter running under a custom host in this case. It's a separate process from the debugged app, and it kinda expects to be run using a regular interpreter. But it then applies "python" from launch.json to start the debuggee (and only falls back to sys.executable if it wasn't specified).

[puremourning]

What's not clear to me is why the adapter running under a custom host in this case

we're calling import debugpy; debugpy.listen() from within a host program that embeds the python interp (in my case, the host program is vim). Then debugpy is shelling out to start another interpreter using sys.executable

applies "python" from launch.json to start the debuggee

This is an "attach" scenario.

[int19h]

Ahh, that makes perfect sense now. Yes, we should just make this configurable.

[int19h]

What we want here is really the same thing as "python" config property in the launch scenario, except it needs to be applied before the client has a chance to connect and transmit the configuration in the "attach" request. That's exactly why we added debugpy.configure() - to handle "subProcess" in the same manner.

So we should just support "python" in configure().

[dalmago]

A possible workaround for Windows is to use sys.exec_prefix and concatenate it with \\python.exe. I'm replacing sys.executable temporarily, before calling debugpy.listen. As said above, "it's absolutely not art".

[ghaefele]

I have the same problem with ArcGIS Pro and Python Toolbox Tools.
The adapter times out ("timed out waiting for adapter to connect") whatever I try (changing sys.executable, attach using listen, attach, attach using process id...).
If I don't change sys.executable the debugger starts a new ArcGIS Pro instance.

Until June 2020 I used ptvsd with "attach" and that worked fine.

Environment:

• Windows 10
• ArcGIS Pro 2.6 with embedded Python 3.6.10 (custom Miniconda environment)
• VS Code 1.48.2
• debugpy 1.0.0rc2

[daniele-niero]

A possible workaround for Windows is to use sys.exec_prefix and concatenate it with \\python.exe. I'm replacing sys.executable temporarily, before calling debugpy.listen. As said above, "it's absolutely not art".

This is not gonna work for those who are using a python embedded in certain applications, for example in Autodesk Maya, where the python to execute is actually named "mayapy.exe" and mayapy.exe is not even in sys.exec_prefix...

Speaking about Maya... this issue is blocking the entire VFX and Gaming industry from using debugpy and consequentially Visual Studio Code for debugging.
Unless there are some other workaround I'm not aware of.
No pressure to the developers ;)

[Felk]

Unless there are some other workaround I'm not aware of.

The workaround is to temporarily mock sys.executable to some existing python.exe, not specifically an embedded one. From the code puremourning linked, modified:

    exe = sys.executable
    try:
        sys.executable = "C:/path/to/any/working/python.exe"
        debugpy.listen( port )
    finally:
        sys.executable = exe

Specifically, it can just be a completely different python installation. I recommend installing python of the same major version on the machines you need to debug, if that works for you.
(Edited the comment to answer daniele-niero's comment, as I originally misread that)

[int19h]

@daniele-niero This was just a proposed default for the case where sys.executable isn't available. The ultimate fix is to allow the path to Python interpreter be specified explicitly. This is exactly what #387 does.

[Hecatron]

Just to mention I ran into this issue recently when trying to see if debugpy would work with freecad (the latest 0.19 one that uses python 3) since it also has an embedded python interpreter
for now I'm just going to try and use ptvsd until it's sorted.

[int19h]

#387 is now merged, and you can do debugpy.configure(python="/path/to/python") before debugpy.listen() to designate the Python interpreter that should be used by debugpy to spawn its helper processes.

[picrots]

I had this issue with Maya and after spending few hours reading all these comments I ended up with 2 solution:

1- using debugpy module and set python in config method:

import debugpy
debugpy.configure(python="<path_to_maya_python>/mayapy")
debugpy.listen(5678)
debugpy.wait_for_client()

2- using old ptvsd

import ptvsd
ptvsd.enable_attach(address=('localhost', 5678), redirect_output=True)
ptvsd.wait_for_attach()

I really prefer it to work out of the box like in ptvsd.
Using debugpy means that I have to remember and type mayapy path each time I want to use maya. (yes I can copy and paste from somewhere but this is just a layer of unnecessary friction)
But more importantly for each new Maya user there would be some time wasted to make it work since they can't find anything regarding it in official documentation.

I am just wondering how difficult it is to implement\transfer the same behavior from ptvsd to debugpy?