Migrating VR Libraries ft. GSoC 2023

[p4puniya]

Introduction

Hi everyone,

This is an open discussion for implementation of *Migrating the VR libraries from GVR to Cardboard SDK in processing-android as part of GSoC'23(extended timeline).

Lets start with Cardboard SDK - You can use the Cardboard SDK to turn a smartphone into a VR platform. A smartphone can display 3D scenes with stereoscopic rendering, track and react to head movements, and interact with apps by detecting when the user presses the viewer button.

What's the need to switch to Cardboard SDK?
Since 2019, Google stopped supporting the Google VR and deprecated it. Instead, it shifted it's focus towards native libraries which was already in use: Cardboard SDK. Due to this, Processing's VR library became redundant. Thus, by migrating to Cardboard SDK, we can make the VR library functional again.

After a thorough discussion with @ranaaditya , we created a rough idea on how we can achieve this:

• Add C++ support to Android Mode by editing the build gradle and VR templates.
• Import the Cardboard SDK, NDK lib and .so files(eg. arm64 , armeabi, x86, x86_64)
• Create a JNI bridge, CMake list and other c++ dependencies.
• Change the existing VR templates to add external native builds
• Map the new methods( from Cardboard-SDK) to their predecessors( from GVR).
• Migrate the methods to the new SDK.
• Test the new library with sample apps.

This will be an exciting project for me and everyone is welcomed for active participation in this discussion!

References
Adding JNI to an existing Android App
Link Gradle to your Native Library
Adding C/C++ code to our project

What are your thoughts on this @codeanticode, @ranaaditya ?

Thanks,
Gaurav
GSoC Contributor: Gaurav Puniya

[ranaaditya]

Thanks @p4puniya for this discussion.

Hi @codeanticode , me and @p4puniya have been discussing and going through the cardboard sdk from almost a month. Gaurav as of now have gone through the cardboard sdk codebase and trying to import the sdk into processing, adding a jni bridge for it.

Thanks,
Aditya

[p4puniya]

Updates on Current progress.

• I've successfully changed the VR templates to change the sketch directory to support JNI. The updated template includes Cardboard SDK copied to app folder, Jni library copied to a newly created JNI folder inside Main folder, CMakeLists.txt being copied to app folder.
• Since we need the project to support NDK, we had to make changes for the app to install it separately inside user's SDK path, in case none is found. I made the changes so that P-Android-VR checks whether any NDK is already present or not. In case not, It installs NDK version 21, as that's the Version most compatible with the Current PDE+Cardboard build.
• I've also added the misc. files that facilitate the support of cardboard such as arm64 etc. Though some were already present, I made sure to check for missing ones and then copy them.

Currently, I'm trying to add the JNI bridge but facing some issues about how I should go about it. Usually, the JNI targets the MainActivity, and these activities use the native functions. But processing works very differently. The Java files such as VRActivity.java, VRGraphics.java etc are solely responsible for constructing the VR app in sketch. Though @ranaaditya is helping me figure things out and work accordingly, I would really appreciate the participation of @codeanticode and other community members in this discussion, since this project is fairly complex and could use the expertise of fellow community members.

Thanks.

[codeanticode]

@p4puniya this sounds like great progress!! What are the specific issues you are having with the JNI bridge? I'm happy to take a look at this.

[p4puniya]

Hi @codeanticode . So now we are trying to build the JNI. Cardboard has a single VR activity (Main Activity) and is using the JNI bridge to point the native function there.

Like this:
'''
#define JNI_METHOD(return_type, method_name)
JNIEXPORT return_type JNICALL
Java_com_google_cardboard_VrActivity_##method_name
'''
Now, if We look at Processing VR, then instead of the Main Activity, we have multiple Java classes such as VRActivity, VRGraphics etc. So, I'm a bit confused about how we should proceed on for adding the JNI bridge. Should there be a single file for all the Native functions and then these classes importing and using that file to use the native function? Or should the JNI point to all these classes so that the native functions can be used directly in them?

Please guide me on this.
Thanks.

[p4puniya]

Brief about current progress:

I want to brief everyone about the approach we have considered and worked towards until now, as well as the problems/challenges faced in each approach.

Task at hand: Migrate the GVR libraries to cardboard sdk.

Approach 1:

Change the templates of the final VR sketch to include Cmake, Cardboard NDK, and change VRActivity template to directly call the functions and build the VR app.

• Using this approach, the VR App will be build successfully. (Quite similar to the sample app provided by Cardboard NDK: here )
• But this approach has a major flaw to it. It completely neglects how Sketches are build and doesn't use the Processing files such as Sketch or Papplet and completely relies on Cardboard NDK. This will just be like developing a normal Cardboard App without using processing and thus kills the purpose of VR integration in processing android.

Work Done:

• After discussing this approach with @ranaaditya and we came to the conclusion this is a bad solution to the problem. But we could use this idea to come up with a better one that keeps Processing Android in the loop.

Approach 2:

We brainstormed on how to integrate the previous approach with the current Processing VR. After much research and discussions, we came up with a solution, that we can set up the Cardboard SDK , then create a JNI bridge between the Java files (Such as VRActivity, VRSurface etc. here ). And integrate the cardboard NDK with Processing Android. We started to implement this approach but were faced with many challenges:

• Processing Android worked on different versions of gradle, tooling, and other dependencies compared to Cardboard NDK. To make them work together, I had to run multiple tests and do many trials to make sure everything could be compiled and nothing breaks.
• The templates for both, Android Build and VR-Gradle needed to be changed as we had to add CMakeLists.txt, complete cardboard SDK, JNI files, supporting libraries etc to the sketch app.
• The final compiled sketch app was incompatible with Native development. When doing native development, Android Studio( or any other IDE you are using) takes care of this internally by adding the NDK support to the app. We had to automate the process of installing the NDK via gradle files and templates after the sketch app is compiled to make sure that NDK and C/C++ libraries are installed and the final android app that's generated using sketch was compatible with Native development.

Work Done:

• I made sure that everything compiled and all the dependency versions were changed, when and wherever required. Though this was a tedious and time consuming task, it was a necessary step to move to our main task.
• I changed all the templates, added the CMakeLists.txt, Cardboard SDK, JNI Files and other supporting library. This resulted in a massive Code base being added to the processing android.
• After many failed attempts, I was eventually able to add the Android NDK to the sketch app. Apart from this, I also had to add Cmake functionality and add a few extra .so files such as arm64, armebi_x86,x64_86 etc, to make sure Cmake worked.
• Currently, the sketch app builds and installs successfully after all these changes. (The code changes were quite big, almost 177 files were changed with 23k+ additions as Cardboard, NDK, and various other libraries were added. here ).
  After this we decided to move to adding the JNI bridge. But as I begun with that I was hit with another road block. To construct the JNI bridge, the Java files were needed to be compiled at runtime, along with CMake, C++ files and other supporting libraries with the use of gradle. Now this posed quite a problem, as the Java files( VRAndroid, VRSurface, VRGraphics etc ) were standalone files whose code base was being copied directly to a jar (VR.jar) without compiling, and this jar was integrated in the final sketch app. Due to this, we couldn't make the JNI bridge.
• After some research online and a long discussion with @ranaaditya , I found that a possible solution could be to compile a fat JAR by creating the JNI before the jar is compiled and then send the Cardboard SDK, Java Files, JNI libraries and other supporting code as a fat jar to the sketch app and then use that.
• This led us to the third Approach for this task.

Approach 3:

• We planned to add the JNI files at the base level, compile the files and then send them as a fat JAR to the sketch app. Now this is a challenging task as we found close to no similarity between how Google VR and the current Cardboard's internal working.
• Also, while trying to incorporate the JNI bridge, we found that the Cardboard SDK was build to work in an Android Environment and failed miserably when trying to be added as simple C++ library. It relied heavily on Android.jar and other supporting dependencies. I tried to add them externally, but am yet to completely integrate them.
• I decided to create some helper functions, such as helperViewPort, helperEye, helperFieldofView, that would work in ways similar to how GVR worked but relied on the C++ libraries for it's internal working. Currently we are trying to add the VR Renderer, but Google VR and Cardboard SDK have completely different internal working for renderer, headtransform, eye, distortion mesh etc.
  To over come these issues, I need guidance from someone who knows Processing Android VR in and out and can help me setup the renderer and JNI for the same. Thankfully, @codeanticode has decided help me on this via a zoom meet and we will try to resolve this issue.

This approach has already resulted in close to 1000 additions in the code base which are mostly self implemented. ( here )

Another 4 ( Not implemented yet):

• I came up with this approach last night after constantly working on this project for more than 28 hrs straight, so this one is yet to be discussed with the mentors.
• As the Cardboard NDK works well with compiled gradle files and current processing Vr is working without compiling the Java files, we can create a simple Processing VR from scratch, that uses methods that call functions from VRMainActivity after the sketch app is compiled to call methods such as renderer, display metrics etc, and send them to the VRGraphics or VRSurface java files, which send them to sketch.
• This seems to be the ideal approach which utilises everything mentioned above and has never been implemented before. Though, the current time constraint of GSOC doesn't allow me to work on this as it would require building everything from scratch(templates, gradle files, base java files, Androidbuild, etc). And this would need constant guidance from someone who has build a complete android library before.

I do hope that this clears up everything I've worked on until now, but feel free to ask any doubts or anything that comes to your mind.
Thanks.
Best Regards,
Gaurav.