Implement readPixelsToTemporaryFile

[gre]

https://github.com/gre/react-native-webgl/blob/master/src/RNExtension.js#L20-L29

this method is not yet implemented on the native side but is needed to implement (efficient) snapshotting. (see also gre/gl-react#93 )

that method takes:

x: number,
y: number,
w: number,
h: number,
options?: {
  format?: string,
  quality?: number
}

[chachasikes]

I can help in whatever way I am able (which is limited by my experience & time)

But to introduce myself: I'm using expo & gl-react-expo (switched over from native gl react v2 - and upgraded to v3 & following this issue of capturing gl surface data)

A couple of months ago, I managed to capture some surfacedata (for v3) using an inefficient method as a temporary fix for personal use of an app (UPNG encoded surface data base64 saved using imagestore savetocameraroll) ... 4000x4000 px images crash it, but 500x500 was making it through on iOS simulator. And then I botched it up on trying to publish it & upgrading packages... and then I had to put it away.

I'm considering disconnecting from Expo til I can solve this & trying to make a native module so I can use the PNG saving feature that might be more efficient... if so, I'll try this.

I'm a little bit concerned about the iOS11 new file system changes that I don't know much about, but plan to upgrade soon. So I am reading forum posts trying to catch up on what's happened lately.

I will definitely follow this issue, and if i make any progress - share what happened -- or try/review any code anyone else might make.

Thanks! I can't wait to see this all come together when it does!

[kesha-antonov]

@gre @chachasikes Hello! I have some time to investigate it and I want help as need this feature too.

Can you give start tips, docs, examples where to look? Or share some code? Even if it's not working perfectly.

[gre]

the main difficulty is probably to figure out how this will works with the current C++ implementation, as a method there. I think it needs to be a method so it is executed as part of the queue (like if you do readPixelsToTemporaryFile after a drawArrays but before another one, it needs to effectively retrieve pixels from the previous drawArrays..). Ideally, it also needs to be non blocking and return a javascript Promise. finally I think there will be some iOS and Android specific part to write (because the conversion part / format support depends on platform as well as the "file saving to a tmp file" & we probably don't want to reinvent the wheel on c++ side).

so to summarize I think we need:

a new gl method in C++, that returns a JavaScript Promise (or maybe it's a callback, and we reconciliate on JS side). this method needs to alloc an array and do a gl.readPixels() and then give that array to the iOS/Android impl that will (in async) returns back an URI.

the difficulty is there is not yet any example in an existing C++ method to do that, so there will be some R&D on this.

and maybe there is a another simpler way to implement this but that was my 2-cents

cc @nikki93

[gre]

another maybe simpler idea is maybe you would call something via the bridge (kinda like in TextureLoader) and you call some C++ code that will stack something into the gl queue. still will need a callback mecanism

[kesha-antonov]

Ok. Background/async + promise/callback is not a problem. I've done similar task in other lib

Any examples of this: alloc an array and do a gl.readPixels()?

[gre]

I don't have yet a good knowledge on the architecture decision of the current WebGL.cpp code

the tricky part about async/callback is not really on the iOS/Android bridge side, but more on how this will works in the current C++ code. Like I don't know how to give a callback to cpp, especially for Java Android impl 🤔

that's why i'm not sure how this should be implemented (like should it be a call to a native bridge code that then would call the c++ code VS should it be all in c++ and maybe something to call from there). what's for sure is there will be some code to be done on both side, i'm just not sure how some part can communicates. (I know how to communicate Objc/java -> cpp, there are a few examples of that, but i'm not sure about the other way).

also maybe the array is allocated by the objc/java side but will still need a way to listen to "ok it's filled now" then you can write that into a file, get a url and and release the array & finally promise.

sorry if it's a bit confused but that's my direct thoughts :D if you want I can try a PoC, even if uncomplete would probably help

[gre]

I think these can help

https://stackoverflow.com/a/13397357/343892
https://stackoverflow.com/a/28689902/343892

still are some raw ideas, so need to think

[kesha-antonov]

Ok. I thought that allocation will be in objC. It's cpp. I did work with java/objC callbacks https://github.com/shahen94/react-native-video-processing/blob/master/android/src/main/java/com/shahenlibrary/Trimmer/Trimmer.java#L168

So we call JS -> objC/java -> cpp (allocate and get pixels from gl node) -> objC/java promise -> JS uri of new img

Right?

[gre]

actually yeah alloc may be done in objc/java , need to see which makes the most sense, but in both case, at some point I think you need to communicate back so the promise is released. like, cpp needs to tell "hey iOS, that pixel array is now filled, do the work!" (which maybe is just a function without param)

yeah I think you get the pipeline correct :)

[gre]

on Java side it might be a bit more hairy but I managed once to have Bitmap -> pixel array working:

https://github.com/gre/gl-react/blob/v3.10.0/packages/gl-react-native/android/src/main/jni/GLImages.c

(this code was a workaround to get pixels from bitmap sent to the cpp, so it can draw things with an id from a texImage2D)

so maybe Java just instanciate a bitmap with correct size and then give that to cpp that will fill it in the future (not sure)

[gre]

@kesha-antonov I have started something in this: 2f31cad

adding a C method void RNWebGLContextReadPixelsRGBA(RNWebGLContextId ctxId, GLuint x, GLuint y, GLuint w, GLuint h, void* dataToFill, void(^onDone)(void));

however there is something i'm not sure about, this onDone callback.. there is currently an issue that my iOS block gets somehow unallocated and at runtime it will EXC_BAD_ACCESS – except this part, I think the code should work (only started for iOS, probably going to try Android now).

there is a new example app (the example folder) that have a CAPTURE button to test these.

[gre]

I also tried to copy the block at difference places but does not seem to help 🤔

I'm not an expert of ObjC/C++ 😀

[gre]

I also have some doubt this can even work out, I mean, a callback from the GL/c++ thread to the ios main thread? isn't this the issue?

[gre]

I think I need a more complex mecanism anyway because Android C does not even seem to support ^{} blocks (it's a C extension)

[gre]

probably need to rethink this architecture with some sort of registry on objc/android side so you store some tmp object that eventually get restored with an id..?

[gre]

Discussed a bit with @nikki93 today and I think there is an even better solution raising now:
let's solve this problem in another lib that would takes a pixel typed array and generate a file. The typed array allocation shouldn't be too expensive, readPixels will have to be blocking on the GL thread anyway so probably we are shifting to this kind of API:

pixels=new UInt32Array(w*h*4)
gl.readPixels(x,y,w,h, gl.RGBA, gl.UNSIGNED_BYTE, pixels)
SomeLib.encodeRGBAImagePixelsToFile(pixels, w, h).then(uri => console.log(“image”,uri))

this is somehow more standard and also I think is more powerful (more loosely coupled) & could technically be used for something not even GL related.

there is no work to be done for this current library because already offers gl.readPixels.

NB
In this other lib, one important thing will be figuring out how to have typed array support in react-native, but it's not impossible to solve as proven by the current library that have typed array support.
it would be trivial if facebook/react-native#1424 was fixed, hope the workaround is not too tricky.

[kesha-antonov]

Very interesting investigation!

[kesha-antonov]

Next task is to create external library with JS + objC/android with async method encodeRGBAImagePixelsToFile ? @gre

[gre]

yes. the only tricky part is figuring out how to support TypedArray, might need to write some C++ because it is the way to access the lowlevel JSC runtime and do this work. that's the trickiest issue that this library will have to solve because FB don't have typedarray support yet facebook/react-native#1424