How to use python 3.x for scripting Ghidra?

[dgutson]

Hi,

since Jython is mostly abandoned, what would it take to replace it by py4j? The approach is different.

[ryanmkurtz]

Looks like it's providing an interface to talk between Python and Java over a socket. Probably not the ideal solution for Ghidra, but if someone wants to make a proof-of-concept it might be worth considering.

[dgutson]

I might get someone to try it. Any clue about where to start? Is there an internals implementation document?

[ryanmkurtz]

The Java side of things should most likely be a Ghidra Plugin. The best documentation we have for it is referenced in #4240. The hard part will be creating a GhidraScript-like environment like we have done with Jython (where script state is injected into the Python environment so you can access variables and methods as if inherited from GhidraScript)

I also have security concerns with opening a socket, so that will have to be investigated too.

[ryanmkurtz]

NOTE: The goal would be a proof-of-concept to see how it compares to our current Jython capability. We can't really give any guarantees about whether or not we'd want to move forward with it after the proof-of-concept stage.

[dgutson]

The motivation is python 3.x support, which Jython doesn't support. We tried the ghidra bridge but it is too slow.

[ryanmkurtz]

I'm also hopeful that GraalVM will have good Python integration one day.

[dgutson]

Didn't know about GraalVM, we just took a look, supports python 3.8, seems a much more interesting approach. Researching.

[ryanmkurtz]

@dgutson I haven't had a chance to play with this extension yet but you might want to check it out:

https://github.com/jpleasu/ghidraal

[hedgeberg]

I would note that the problem of python 3 in ghidra, specifically, is already solved by another government entity: https://github.com/dod-cyber-crime-center/pyhidra

Leverages jpype to integrate native python installations into ghidra, and allows for importing ghidra modules into native python. I suspect that this is a better solution (at least in the near term) than py4j or graalvm.

[Thrameos]

To see a list of options look over. https://jpype.readthedocs.io/en/latest/userguide.html#alternatives

For what it is worth jpype is currently DOE code as I am one of the main authors. Is there specific security concerns that you need assistance with?

[hedgeberg]

Is there specific security concerns that you need assistance with?

@Thrameos to be clear, it's not that there are security concerns -- ghidra currently leverages jython to enable python scripting, but jython is dead as disco so an alternative to jython is being suggested. Someone over at DC3 implemented a ghidra/native python bridge using jpype that seems like a good alternative.

[ryanmkurtz]

Is there specific security concerns that you need assistance with?

No, just a reminder to understand how the interprocess communication works to make sure there are no concerns.

[TorgoTorgo]

Didn't know about GraalVM, we just took a look, supports python 3.8, seems a much more interesting approach. Researching.

I have tried Graal VM (modified Ghidraal), and got python3 working but unfortunately support for python packages is poor.

I would recommend investigating another approach for Python3 until Graal improves compatibility.

[ryanmkurtz]

Thanks, I've never had the time to look into it, but that was my understanding as well.

[Thrameos]

@ryanmkurtz JPype uses JNI for direct communications. There is a detailed developer guide that lays out each piece of the communication. It is all CPython so it looks like native Python heap types. JPype is currently a forward bridge only as it only exposes Java to Python not the other way arround. It likely has the highest level of integration as it is meant for speed in scientific coding. If you have need of a forward/reverse bridge I think the jep is currently your best option unless you want to provide some aid to the stalled epypj project. Py4j is for heavy wrapper backends only as it has no direct communication with Java and limited transfer capabilities, but has the advantage that multiple JVM can be controlled from one Python process. All other wrappers are dead as far as I am aware.

JPype wrappers for Java can either be heavy or light. Heavy wrappers hide Java classes entirely (basically uses jpype as a back end). Light wrappers use the customizer system to just add pythonic interfaces over Java classes to make them work like Python types. Currently I haven't implemented any security model such as preventing monkey patching to make classes closed. If you wanted really secure packages I would need to implement the model in which the customizers are stored in the jar directly such that a verified file holds all pieces. Noone has really needed that so it has always been backburnered and usually the Python devels and Java devels are separate entities so shipping the Python pieces in the Jar is a no go.

Transfer of large amounts of data uses Python buffer protocol wrapping Java ByteBuffer. There is not currently a native Java InputStream or OutputStream to Python IO though that is not hard to write. I will happily accept requests /improvement ideas on the jpype github issues list.

Jpype is technically matured software as its has active developers, but not a lot of remaining features to implement beyond the reverse bridge where I can't find a critical mass of users to help support. I will usually implement any reasonable request so long as it doesn't rise outside incentental tool development exception as per my DOE contract and someone presents a reasonable use case. Though as my workload is pretty high due to manpower issues (curse of physicists getting poached as software devels in the bay area), sometimes it takes so poking to keep feature requests on track.

The lab has finally signed the Python contributor agreement so I can contribute improved hooks to Python code base. I tried contributing to jython to get it up to scientific Python3 code needs, but basically got turned down so I focussed on jpype instead.

[ryanmkurtz]

Thank you for the detailed information @Thrameos

[ryanmkurtz]

@dgutson, having learned about ghidraal and pyhidra, would you like to keep this issue open? If so, it would probably be appropriate to retitle it to represent its new goal.

[dgutson]

@ryanmkurtz @Thrameos pls check dod-cyber-crime-center/pyhidra#15
We can't use pyhidra as-is yet.
I agree with the retitle.

[Thrameos]

It seems like most of what you are looking for is already available in pyhidra. However, if you do need specific modifications or wish to help with reverse bridge capabilities (so that a module developed in Python can be used in Java), then feel free to ping me.

[mike-hunhoff]

It's really great to see this discussion open around using Python 3.x scripting capabilities in Ghidra! Today we released another solution, Ghidrathon, that uses the open-source project Jep to embed CPython in the JVM. Ghidrathon supports an interactive Python 3.x interpreter window, integration with the Ghidra Script Manager, and Python 3.x script execution in Ghidra headless mode.

Ghidrathon installs as a traditional Ghidra extension allowing you to use Python 3.x without modification to Ghidra or special execution of Ghidra. We use Jep to start and configure CPython from the JVM and inject script state into the Python environment similar to how the existing Python 2.7 extension works with Jython (in fact most of Ghidrathon's code is based on the existing Python 2.7 extension). This provides Python access to variables and methods as if inherited from GhidraScript (see example Ghidrathon Python 3 script here). By developing Ghidrathon this way we hope it can serve as an example of using Java (and Ghidra's extensive APIs) as a front-end for back-end Python 3.x processing.

We'd love to hear any feedback from those who have a chance to use Ghidrathon. You can also check our blog post for more information.

[ryanmkurtz]

Thanks, I look forward to trying it!

[dgutson]

@mike-hunhoff great! Could you please share a few lines about how Ghidrathon compares to Pyhidra? Eg a criteria to choose one or the other.

[ryanmkurtz]

I just tried Ghidrathon and was quite impressed! I only used it for a few minutes, but I was able to use the scripting API without needing to import anything, and I was able to stray outside the scripting API into more advanced areas of Ghidra with the proper imports. I was not able to import standard java libraries like you can with Jython...I'm not sure if that's possible with this setup.

I am going to recommend that more people on the Ghidra team try it so they can weigh in too. Great work!

[mike-hunhoff]

@dgutson it really comes down to the underlying implementation. Ghidrathon uses Jep to expose Python to Java. This allowed us to develop Ghidrathon as a standard Ghidra extension that, once installed, does not require modifying Ghidra or executing Ghidra in any special way. This includes executing Python 3 scripts in Ghidra headless mode which benefits existing pipelines as no changes are needed to start using native Python 3. Overall, our goal is to provide Ghidrathon users a seamless transition from the existing Python 2.7 (Jython) extension. Pyhidra uses JPype to expose Java to Python which requires special configuration and execution of Ghidra. This may be more prone to breakage as future versions of Ghidra are released.

[mike-hunhoff]

@ryanmkurtz thank you for your feedback! You should be able to import standard Java libraries as demonstrated below. If you have specific/different cases we can take a closer look.

[Thrameos]

@mike-hunhoff I am a little confused as to what special configuration would be required for using a Java library with JPype. Is it something to do with Ghidra? I haven't heard of any other library requiring modifications.

[mike-hunhoff]

@mike-hunhoff I am a little confused as to what special configuration would be required for using a Java library with JPype. Is it something to do with Ghidra? I haven't heard of any other library requiring modifications.

@Thrameos normally you run Ghidra using the appropriate ghidraRun script but pyhidra requires starting Ghidra using pyhidraw. I am not overly familiar with JPype or pyhidra's internal workings but I believe using this command invokes pyhidras launcher code used to configure an environment in which JPype can connect to Ghidra.

[LittleNewton]

@mike-hunhoff Hi, Ghidrathon is a wonderful tool! Now I'm using it to develop a analysis tool in VSCode IDE. However, I don't know how to debug with Ghidrathon. Is there some learning materials?

[aur3l14no]

Maybe just use breakpoint()?

[LittleNewton]

I have given up Ghidrathon. Using official Eclipse-based SDK is much easier to debug. Java is still much better!

[aur3l14no]

Ah, after testing, I find out that breakpoint() does not work. Yeah maybe Java is the way to go.