How to debug a Ghidra Python script from Eclipse?

[pasccom]

Hello, I do not manage to set breakpoints in a Ghidra Python script from Eclipse (with PyDev and GhidraDev). Is this really supported or did I misunderstand?

What I have done so far:

• Installed the latest Eclipse version (Version: 2020-12 (4.18.0), Build id: 20201210-1552)
• Installed Ghidra 9.2.2 from zip file on official website
• Installed PyDev (8.1.0.202012051215) by adding the PyDev update site to Eclipse installation sites
• Manually installed GhidraDev (2.1.1.202009301453 for Ghidra 9.2.2)
• Configured Eclipse installation directory and workspace in Ghidra
• Configured Ghidra installation path and Ghidra Jython interpreter in eclipse

I start Ghidra from Eclipse using (Debug As). For testing purposes I have made a simple script which only prints some variables and set beak points on all the print lines and the lines with function calls:

from ghidra.python import PyDevUtils
from ghidra.util import SystemUtilities

def test():
	print 'test'

if (__name__ == '__main__'):
	test()
	print PyDevUtils.getPyDevSrcDir()
	print SystemUtilities.isInDevelopmentMode()
	print SystemUtilities.isInHeadlessMode()
	exit()

It runs well and I get in Ghidra console:

test
/good/path/to/pysrc
False
False
SystemExit

In the debug pane, I see that the debug server gets a connection and that threads are created under ghidra.GhidraLauncher, but no breakpoint is hit.

Did I overlook something? Is this unsupported? Anyway, thank you for any help you could provide. I could do without a debugger, but it would be very helpful to have one.

[ryanmkurtz]

This issue has been reported in an unrelated old ticket. We can use this new ticket to focus solely on the debugger not connecting.

An important thing to note from the old ticket is:

I have opened an issue with PyDev to address the breakpoint not catching in newer versions of PyDev: fabioz/PyDev.Debugger#171

Unfortunately, it hasn't gotten any attention yet. The breakpoints work in PyDev 6.3.1 though if you are willing to use an old version.

[jpleasu]

as a workaround, you can try add the following to your code (run as early as possible). It will depend on your version of pydevd.py

import pydevd;pydevd.threadingCurrentThread().__pydevd_main_thread = True

or

import pydevd;pydevd.get_global_debugger().has_user_threads_alive=lambda:True

or

import pydevd;pydevd.get_global_debugger().has_threads_alive=lambda:True

[pasccom]

This issue has been reported in an unrelated old ticket. We can use this new ticket to focus solely on the debugger not connecting.

Yes, I think it is better that we open a new ticket, because I had the issue (I actually had installed PyDev inside a .eclipse hidden directory in user home directory, since eclipse could not write in the plugins directory), saw this ticket, ensured Eclipse properly installed PyDev into the plugins directory and did not read any further, since I thought I had addressed the root cause of the issue. Thanks for pointing me back there.
I now downgraded to PyDev v6.3.1 and the breakpoints work, as long as they are before the askChoice() function (and certainly other user interaction functions). I also tried single-steping over the askChoice() function and the program does not stop anymore. I am not sure whether it is worth creating a separate issue. But I can do it if it is necessary.

On this topic, I can see something strange in the threads of GhidraLacher. When I hit the breakpoint before the askChoice() function, I have the following new threads:

whereas as soon as I step over the askChoice() function the pydev.* thread disappear and I am left with:

Thanks @jpleasu, but with PyDev 6.3.1 it does not change anything (EDIT: I tested the import pydevd;pydevd.get_global_debugger().has_user_threads_alive=lambda:True). I am too lazy to try with newer PyDev versions.

[jpleasu]

I checked the source for 6.3.1, and the __pydevd_main_thread workaround should work.. around... at least some of the issues pydev has with daemon user threads. The has_threads_alive variant should prevent more cases where the check thread kills the client.

You may want to upgrade.. with pydev 8.1.0 (and the workaround above), I'm not having any issues with breakpoints before and after dialogs.

[ryanmkurtz]

I've confirmed that adding

pydevd.threadingCurrentThread().__pydevd_main_thread = True

to the start of your Python script will make the breakpoint catch using Pydev 8.1.0. Nice find @jpleasu!

This is something that should be added to the below code so the user doesn't have to worry about adding that line themselves.

ghidra/Ghidra/Features/Python/src/main/java/ghidra/python/GhidraPythonInterpreter.java

Lines 218 to 223 in 6fd7f6d

	InetAddress localhost = InetAddress.getLocalHost();
	new Socket(localhost, PyDevUtils.PYDEV_REMOTE_DEBUGGER_PORT).close();
	Msg.info(this, "Python debugger found");
	exec("import pydevd; pydevd.settrace(host=\"" + localhost.getHostName() +
	"\", port=" + PyDevUtils.PYDEV_REMOTE_DEBUGGER_PORT + ", suspend=False);");
	Msg.info(this, "Connected to a python debugger.");

We'll also need to test older versions of PyDev with this change to see if we need to tweak our oldest supported version.

[jpleasu]

Yes. @ryanmkurtz you're welcome to make the change -- the workaround should come before the settrace, otherwise there's a race.

[pasccom]

Thank you very much @jpleasu for the workaround. It works perfectly with both PyDev 6.3.1 and 8.1.0 and I have no problems with dialogs anymore and I am able to connect to a running debug server without starting Ghidra from Eclipse, which allows me to work around the potential racing condition you mention (even though I have never encounter it in my many tests)

While waiting for the fix in the official version, I will use the following function at the beginning of my scripts, it ensures that PyDev tracing function is correctly setup in all cases (running Ghidra from Eclipse or connecting to a running debug server)

from java.lang import ClassCastException
import sys

PYSRC_PATH = '/path/to/pydev.core/pysrc'

def setupPyDev():
	# Ensure that PyDev pysrc directory is in Python path
	if PYSRC_PATH not in sys.path:
		sys.path.append(PYSRC_PATH)
	
	import pydevd
	pydevd.threadingCurrentThread().__pydevd_main_thread = True
	
	# Ensure that PyDev is tracing
	try:
		sys.gettrace()
	except ClassCastException :
		pydevd.settrace(suspend=False)

Thank you both for your help

[ghidra1]

Change has been merged into patch and master branch for release with 9.2.3