WebGL "shim"

[nickdima]

Sorry, if this is a bit out of the scope of the project but I was wondering if it would be possibile to build a "shim" for React Native that would let one transparently use OpenGL as using WebGL in the browser, and how to go about doing this?
The goal would be to use libraries like pixi.js (maybe trough react-pixi) for building universal components that render both with WebGL and OpenGL.

I've posted a related question also here: pixijs/pixijs#2120

[gre]

This is a huge challenge.
not only you have to implement a mapper from your WebGL shim to OpenGL (the API is huge and there is a lot of differences),
but also you will have to figure out how this can work through the React Bridge (communication between the JS thread and the UI thread). This second part might be difficult to make it performant, especially when using WebGL, you use a lot of typed array, and there is currently no support for sending a typed array in React Native ( see facebook/react-native#1424 ).
Also it sounds like you will have to reimplement Image() if you want to shim all code that uses textures.

That said, I would love to see a react-native-webgl happening, but this is a bit out of the scope of gl-react-native as you said (right now, gl-react-* focus on a subset that just allows you to write a fragment shader and it's already challenging).

[nickdima]

Thanks a lot for your answer! Since I don't have experience with OpenGL, I now got a clearer picture of the work involved for pulling this off.

[lwansbrough]

@nickdima GLKit uses OpenGL ES 2.0, which is the same thing WebGL uses (though I suppose there could be a few differences.)

Would this not "just" be a case of mapping the HTML 5 canvas WebGL context to the native OpenGL ES 2.0 implementation? Android and iOS both support OpenGL ES 2.0.

Typed arrays can be represented by regular arrays and casted into whatever you need on the Objective-C side. Might be a bit of a performance ding though.

Obviously there are considerations to make about performance across the bridge - but we'll cross that bridge when we get there, literally.

[gre]

@lwansbrough sounds like you can start making a react-native-webgl :) It's really a different focus from current gl-react-native path.

gl-react-native's focus is to have the thiner possible layer with OpenGL ES 2 but not exposing the GL API to the JS side.

I guess you will also need to wire all the "gl commands" in JS <> ObjC . And this might be a complex challenge. The WebGL API is not asynchronous by design, you will reach a lot of problem with implementing all the synchronous APIs.

[lwansbrough]

@gre Oops meant to tag you. Yeah I agree it definitely isn't the job of this library, just wanted to pick your brain a bit.

Indeed the synchronous nature of these libs presents a challenge.

[nickdima]

@lwansbrough I posted the initial question because I was building a small game with React (just with plain CSS and HTML) and I was thinking that it would be cool to have a way to build it for both the web and mobile while taking advantage of both React and OpenGL.
I don't have any experience with OpenGL so I was just doing some research :)
What would you use this for?

[lwansbrough]

Just generally having the ability to use 3D JS libraries like ThreeJS would open up a lot of doors for development with React Native. VR and AR via cardboard come to mind.

[nickdima]

Sounds good

[gre]

Here is some thoughts on the way I think this could be implemented:

Dedicated Thread for WebGL & OpenGL

the WebGL JavaScript could be sandboxed to its own dedicated thread. OpenGL requires you to only use a single thread to run the GL Calls, this thread would be dedicated to this job as well as executing the user's JS, that way there should not be any round trip across 2 threads for bridging the WebGL API to the OpenGL API.

On the JS threads subject, you might be interested by @vjeux's thoughts here:
animatedjs/interactive-docs#5 (comment)

The JS that runs in the WebGL sandbox can be optimised and focused on WebGL job: Native Arrays can be implemented there to reach good performance. The sandbox would also polyfill APIs like new Image() in order to get texture in, and that Image would be just dedicated to this wrapping texture (you cannot do document.body.appendChild with it, right :D).
I don't know how it's possible to expose a Native / C API to the JS, I wonder if something like Node.js native extension would be possible in order to expose all the OpenGL ES 20 functions to be called from JavaScript.

(for actually implementing the wrapper, should look at initiative like node-webgl, headless-gl and the library ANGLE)

Possible API

From the RN JavaScript code we could define it this way:

<WebGL
  entry="myWebGL.js"
  width={width}
  height={height}
  onInit={socket => { socket.onmessage = answer => ...;  socket.postMessage("foo"); }}
/>

^ that socket would be a way to communicate between the 2 JS sides, exactly like a web worker.

now I would imagine it would be great to be able to define some "loaders", like possible in webpack so you can do require("webgl!./myfile.js") and setup things correctly (for this entry prop).

also I would expect either a global variable to be available or a function to be explosed in order to get the Canvas and the WebGL Context inside the sandbox.

---

More thoughts need to be done on this, as I might miss some use-cases.

[lwansbrough]

@gre Interesting approaches. I would hope that an API like https://github.com/Izzimach/react-three would be possible. Being able to implement WebGL stuff as a part of the another higher level API should be considered a must have for any implementation. React Three has no DOM dependency, nor does ThreeJS by the looks of it (though you do have to provide a WebGL context).

At first I thought the biggest problem to overcome would be the creation of the API, but now I'm pretty sure it's handling the asynchronous nature of React Native. I wonder if there could be a way to explicitly invoke a native method synchronously (JavascriptCore supports this.) Obviously this feature would have to be implemented into React Native, but it would open the door for modules like this.

[lwansbrough]

There's a bit of discussion around such a solution happening here: facebook/react-native#2212

[lwansbrough]

@gre https://github.com/facebook/react-native/search?&q=nativeTraceEndSection

Synchronous is possible :)

[gre]

cool :) though I wouldn't recommend to block on main JS thread. I think making a dedicated JS thread for WebGL will be an important point if you need to keep 60fps on other stuff.
the question is already raised by RN people just for doing Animated, but WebGL is like 100x or 1000x times more consuming than Animated ;)

[lwansbrough]

Yup, will definitely be on a different thread. I may tinker with it tonight, not sure. Thanks for your help.