Can we create a numpy array from data generated in Java using Javacpp?

[guangster]

Is there a way to obtain the address of a JavaCPP array (e.g. IntPointer) that is usable in numpy in python?

Similar to how ctypes is used for C when a C program output some variable y and a numpy array can be created like

np.ctypeslib.as_array(
    (ctypes.c_double * array_length).from_address(ctypes.addressof(y.contents)))

I am trying to create a numpy array in python from the output of a library running in Java (called from python).
I would like to create a large array in JavaCPP, then access that in numpy in python without having to copy the content of the array to python.

I tried to pass the p.address long pointer in place of the ctypes pointer into numpy but that did not seem to work (e.g. IntPointer p = new IntePointer(new int[]{1,2,3,4,5}); return p; in java, then calling that from python, then passing p.address() ).

Any advice is greatly appreciated!

[saudet]

Yes we can do that. There's a working example doing matrix multiplication using data from Java here:
https://github.com/bytedeco/javacpp-presets/blob/master/numpy/samples/MatMul.java

[guangster]

Thanks for pointing to the MatMul example. I should have clarified, I'm trying to use the numpy object in python rather than in java. My library that is using JavaCPP is running in Java and I have an API exposed to python.

So after studying the MatMul example a bit, I did manage to get this working.
I'm actually quite surprised that JavaCPP supported this!

1. I let my Java library return a JavaCPP object FloatPointer, something like public FloatPointer myJavaProgram()
2. Then, in Python I wrap the returned object like:

    tensor = myJavaProgram() # tensor is a JavaCPP FloatPointer object here
    p = ctypes.cast(tensor.address(), ctypes.POINTER(ctypes.c_float))
    arr = np.ctypeslib.as_array(p, [100000, 443])

3. Now, I have a numpy object arr in python that is using a backing array created by my Java program (no duplicates), which is what I wanted. I can also call tensor.close() to clean up after I'm done with it.

---

I do have a follow-up question. Do you have some suggestions on how we can do this for a GPU array? So have myJavaProgram() return some tensor object that is on the GPU? Ultimately these data will be consumed by pytorch, and my goal is to avoid extra copies of the arrays I'm creating.

[saudet]

If you really want to do it with JavaCPP though, it's still possible. We can create a callback function using FunctionPointer and then call that function from Python using, for example, ctypes: https://docs.python.org/3/library/ctypes.html

[guangster]

Thanks for your insights. Yes, we have been using PyJNIus for methods and classes, but data is too slow this way, maybe because re-creating data in Python is too expensive.

I have been trying a few different options but so far using JavaCPP is by far the fastest and cleanest option, because numpy and torch support creating data from your pointer. It's actually a really great feature for us and we can build models now in python while still taking advantage of flexible and fast Java pipelines.

I know you said JavaCPP was not designed for this but do you have some advice on where this could cause an issue / break down? Is it more of a future compatibility/support issue?

[saudet]

Well, it's just not the same thing. There's a discussion about this at bytedeco/javacpp#17. JavaCPP is designed to create wrappers and bundle native C++ libraries. It just so happens that we can consider CPython as a native library that features a funky script language. :) But like I said, if you're only interested in exposing a couple of things to your existing Python scripts, we can do that just fine with callback functions. Your myJavaProgram() could be defined like this in Java:

    @Platform(...)
    public class MyCallback extends FunctionPointer {
        public @Name("myJavaProgram") FloatPointer call(...) { 
            // ...
        }
    }

And then build that, possibly with a command like this:

    $ java -jar javacpp.jar MyCallback.java -header

Finally, you should be able to load that library in CPython and call that function this way:

    from ctypes import *
    callback = cdll.LoadLibrary("/path/to/libjniMyCallback.so")
    callback.JavaCPP_init(0, None)
    data = callback.myJavaProgram(...)
    // ... 
    callback.JavaCPP_uninit()

Let me know if you encounter any issues with that though.

[guangster]

Thanks for pointing to this! I think the callback is not the big issue here although your suggestion is very instructive of how things work and we may be able to make things better.

What I would like to expose is data from Java to my python script (assuming I already have the part where I call and return Java objects handled via your suggestion or PyJNIus) but without Python seeing the data (because then it is slow).

IMO this is different from bytedeco/javacpp#17 because I am still looking to build a Java program with JavaCPP as a data handler/wrapper.

This is very useful and valuable, because in our typical usecases for machine learning applications, we would like to run train script and experiments in python, but often the data pipeline is in Java. After training, we would like to run a pure Java program (+JavaCPP) running our models in inference mode without any python script.

I will move this discussion to JavaCPP where your link pointed to, as that seems more appropriate.

[guangster]

Continuing here bytedeco/javacpp#530