Error on iOS / Cordova builds

[davemee]

Hi,

We're using this library with great success on Android and macOS Safari, but when we bundle it as part of a Cordova application on iOS we get the following error:

WebGL: INVALID_OPERATION: texImage2D: type HALF_FLOAT_OES but ArrayBufferView is not NULL (k - jeelizFaceTransfer.js:86:267)

This is when testing on an iPad 3 running iOS 9.3.5 (13G36).

Does this error trigger any resonance?

[xavierjs]

Hi,

Unfortunately I don't use Cordova. But I think it has something to do with the webcam access.
This error is normal and handled, it occurs when I test the real WebGL capabilities.

Can you get webcam access using getUserMedia API with cordova?

[davemee]

Hey Xavier,

Sorry about the delayed response - I've been away. A little cursory duckduckgoing (and some simple tests) tells me I don't get the UserMedia API in a Cordova/iOS project, certainly at platform version 4.2 of the iOS platform plugin for Cordova.

There's a plugin for Cordova called Crosswalk which apparently does allow the UserMedia API, as mentioned on this stackoverflow question, but most answers seem to relate to Android, which is accessing the camera fine. I'll post anything I figure out here for anyone else have issues relating to Cordova 7/iOS platform 4.5.5

[davemee]

Hey @xavierjs - an update! (a sad update though!)

Looks like Apple is restricting access to getUserMedia in applications with embedded webviews, which is (clarified with a test case here)[https://github.com/marcusbelcher/cordova-getUserMedia-test]. They're speculating this may be relaxed for ios13, but there's nothing concrete to back that up, sadly!

I went down a Crosswalk route testing out builds on iOS but still could not get this to work, for the reasons detailed above - effectively that Apple are blocking access in a webview.

Sorry to misunderstand the issue and bring it to your attention - but at least hope it may be of use to others.

Thanks for all your help, trouble, and (otherwise) great system, Xavier!

• Dave

[davemee]

If interested, we found this plugin https://github.com/cordova-rtc/cordova-plugin-iosrtc which says it can act as a direct drop-in for the W3C WebRTC API, but we still have the same issues.

[xavierjs]

Hi @davemee
Thank you for sharing.

Unfortunately yes, even in IOS13 Apple does not enable getUserMedia API in webviews :(.
But they still do some (slow) progress, they enabled it in SFSafariViewController ( https://www.linkedin.com/feed/update/urn:li:activity:6562800311763812353 )

I think that before they officially anounce the getUserMedia support in webview, it is not very reliable to use workarounds: either they will rely on tricks that may be broken with some specific versions of IOs or new releases, or they will introduce major bottlenecks slowing down the whole application (I guess they could for example stream the video through a local websocket server...).
Best,
Xavier

[KeiPG]

hello @xavierjs -update
I am a part of the same team as @davemee and we have been working on getting getUserMedia to get webcam access.
We understand you have no experience of Cordova, but hopefully our questions here won't make any assumptions, beyond a general background:
We found a Cordova plugin here , which looks promising. It clobbers getUserMedia, with its own version using native code.
A sample shows this working and displays the camera on our iOS devices.
You'll notice the sample uses the video tag in the html (link).
In our app, we use the same method as the 'cartman' example (thanks!), which creates a canvas, and passes it to the Jeeliz libraries. When we use this method on iOS using the cordova-plugin-iosrtc, we get the same error occuring as before:

WebGL: INVALID_OPERATION: texImage2D: type HALF_FLOAT_OES but ArrayBufferView is not NULL (k - jeelizFaceTransfer.js:86:267)

This would seem to indicate that we are not intercepting the Jeeliz code at the right point.
The Jeeliz library is minified, and we're having difficulty figuring out where to go next: we have verified access to the video stream, via the video tag, but now need to know how that translates to our Jeeliz-enabled application.
We would be very grateful for your help, here - we need to answer the question: how can we help Jeeliz access the video feed from getUserMedia, given that we can access it through the 'video' tag?

Many thanks for your help again :-)

Kei

[xavierjs]

Hi,

This error is normal and handled, it occurs when the lib tests the real WebGL capabilities (see jeeliz/jeelizFaceFilter#85 (comment)).
Maybe there the produced video element cannot be put into a texture.
Can you put the video into a WebGL texture and display it properly?
Can you get this demo working with Cordova? https://threejs.org/examples/#webgl_materials_video_webcam

Best,
Xavier

[KeiPG]

hello @xavierjs
We have been working on trying to get https://threejs.org/examples/#webgl_materials_video_webcam to work on a cordova build and answer your questions. Here is what we have been able to figure out:

"Can you put the video into a WebGL texture and display it properly?" No
"Can you get this demo working with Cordova?" sort of
here is abit more detail

1. Using this code, we were able to get the camera feed to appear on an android device:

navigator.mediaDevices.getUserMedia(constraints).then(function (stream) {
                    // apply the stream to the video element used in the texture
                    video.srcObject = stream;
                     video.play();

2. Using the same example the canvas(with the 3d scene) does not display on the android device.
   We get the following errors, although this is just for information

THREE.WebGLRenderer: OES_texture_float extension not supported.
THREE.WebGLRenderer: OES_texture_half_float extension not supported.
THREE.WebGLRenderer: OES_texture_half_float_linear extension not supported.
THREE.WebGLRenderer: OES_texture_float_linear extension not supported.

3. On iOS we used the same code with this plugin that gives us getUserMedia support. the following code does not work: no errors are reported, no video appeared on the screen

navigator.mediaDevices.getUserMedia(constraints).then(function (stream) {
                    // apply the stream to the video element used in the texture
                    video.srcObject = stream;
                     video.play();

4. The following code does produce video output on iOS:

navigator.mediaDevices.getUserMedia(constraints).then(function(stream) {
      video.src = window.URL.createObjectURL(stream);

This code produces the following error when we try to link it up to the THREE code.

invalid_value teximage2d no video

5. We found the following issue on the cordova plugin repo which seems to indicate that this plugin doesn't provide rendering to canvashere.

we would really really appreciate any help you could give us to direct the video stream from (4) into the jeeliz libraries - any ideas?

Thanks for being there...
Best,
Kei, Alice

[xavierjs]

Hi @KeiPG

Thanx for these detailed informations.
The first step of Jeeliz Weboji, like all our other library, is to update a webGL texture with the video frame.
Then the deep learning engine do render to texture to run the neural network model.
Our libraries don't have fallback without using WebGL. It would be too slow.

Unfortunately if we cannot import the video into a WebGL texture, I can't see how it is possible to use this library with Apache Cordova.

The plugin giving getUserMedia support seems to not be a complete workaround... Maybe you could try to copy the <video> into a standard <canvas> with a 2D context and use the <canvas> as texture source.

But anyway, I think the best way to work on it is to implement a simple WebGL example displaying the video into a WebGL texture (like the THREE.js example). Then it won't be a big deal to adapt this lib to Apache Cordova.

On the Android device, these errors are quite important:

THREE.WebGLRenderer: OES_texture_float extension not supported.
THREE.WebGLRenderer: OES_texture_half_float extension not supported.
THREE.WebGLRenderer: OES_texture_half_float_linear extension not supported.
THREE.WebGLRenderer: OES_texture_float_linear extension not supported.

Because we need either WebGL2, either WebGL1 with OES_texture_float or OES_texture_half_float. The _linear extensions are not mandatory, the lib has workaround if they are not here. What device is it? I think either it is a low end device, either Apache Cordova is not implementing WebGL2 or the WebGL1 implementation is too minimalist and OpenGLES2.0 extensions are not bound. Unfortunately I cannot do a lot of stuff for this.

Best,
Xavier

[KeiPG]

Hi @xavierjs

Thanks for your continued suggestions! We've made a certain amount of progress in getting a webgl canvas available on iOS (before the new version later today :))

We dropped the use of the iosrtc cordova plugin, as it didn't exposed the element to the video tag(it would just display the video in front of the video tag and didn't give access).

We have been trying a few things and have been able to get the camera onto a webgl canvas.
We were able to do this using Cordova CanvasCamera plugin which is able to put the camera feed onto a 2d canvas then using this( this was useful) (this also helps) to put the 2d canvas on a webgl canvas.
we would very much appreciate any further support that could show us how to give the jeeliz library a canvas(as a video element) instead of providing a canvasId.

Thanks Again for being there...
Best
Kei,Alice and dave

[hthetiot]

Hello from https://github.com/cordova-rtc/cordova-plugin-iosrtc
This may help you guys, but it's planned for version 6 after libwebrtc upgrade.

• Related Exposure of mediaStreamRenderers dictionary cordova-rtc/cordova-plugin-iosrtc#253 (comment)

[davemee]

Hey Xavier,

We have a stack of pieces that allows us to render video to a HTML canvas element working in a Cordova/iOS application now. We're wondering if you had any plans to allow the initialisation of Jeeliz to specify a user-provided canvas as the video source on the Jeeliz roadmap?

This is part of a project both being used to teach CS students and to develop an non-profit application providing emotional support for a children's charity, if you would like some positive karma for encouragement!

[xavierjs]

Hi,

I took some days off, but I am back now.
Please submit a pull request with your demo using Cordova (you can put it in a new path demos/cordova for example). I will add the new demo to the lib and add the workaround to take a canvas as input instead of a video to the core.

Best,
Xavier

[davemee]

Pull request issued which will build for iOS and render the camera to a HTML canvas (however, the Jeeliz library is not integrated in this)

Hope that's of use, Xavier!

Best,
Dave

[xavierjs]

Hi,

Thanx a lot for this PR, I will do my best to make it work and keep you tuned here.

Best,
Xavier

[xavierjs]

Hi @davemee,

I cannot get your demo working ( https://github.com/jeeliz/jeelizWeboji/tree/master/demos/cordova/canvas-camera-test ).
I tested with an Iphone XR with IOS 13.0 beta4 and 13.1 and Xcode10.
Then XCode sample app (the single page hello work) works so it is not a problem of permissions/hardware.

I installed latest npm, then Cordova and Ionic using npm.

ionic cordova platform add ios@4.5.5 worked directly.
To be able to build (ionic cordova build ios) I had to install some stuffs:

 sudo npm i -g natives
 npm i ios-deploy
 nano platforms/ios/cordova/build.xcconfig 

and then I commented these 2 lines in platforms/ios/cordova/build.xcconfig :

// CODE_SIGN_IDENTITY = Iphone Developper
// CODE_SIGN_IDENTITY[sdk=iphoneos*] = Iphone Developper

And the project was built successfully.
Then I opened the project with Xcode: open platforms/ios/MyApp.xcodeproj
Everything sounds good. I set the IOS version to 13.0, I plugged my phone and set it as destination.
The App was built successfully. There are still some warnings. The build log is here: https://gist.github.com/xavierjs/444d5fcc074912d869fc3d612362fd38

But when I tried to run it on the device it did not worked. The app was not installed at all. And I did not get any error message. No icon appear on the Iphone, and after a few seconds I have this error message on Xcode:

Could not locate installed application
Domain: com.apple.platform.iphoneos
Code: -1
Recovery Suggestion: Install claimed to have succeeded, but application could not be found on device. bundleId = io.ionic.starter
--
Installed application was not present in database of installed apps after multiple lookup attempts.
Domain: com.apple.platform.iphoneos
Code: -1
--

This is strange because the app launches correctly with the Iphone11 simulator (but I cannot get the camera video stream).

I don't have a Mac, I just borrowed one for these tests, it is not very handy.
I pushed a demo of the raccoon using Cordova here: https://github.com/jeeliz/jeelizWeboji/tree/master/demos/cordova/raccoon
I was not able to test it since I cannot run the basic test on my device. But it should work (after solving some path issues probably). The weboji library has been changed in order to accept a <canvas> or <img> element as videoElement property. If you manage to make it work, I will be very grateful if you could submit another PR.

I have still other stuffs to try (I will test with the latest IOS platform, not the 4.5.5)...

Best,
Xavier

[davemee]

Hey Xavier,

I've been away a few days (and will be again), but I got your app to build and run. However, I'm having a few errors and I'm trying to pin them down. There's a lot in the console generated by Jeeliz firing before things seem to have initialised fully, and something somewhere is disrupting the camera canvas plugin as it's namespace is not registered under the window object - I'll keep investigating and push another PR when I have it running well.

Thanks again for taking this on,

Best

Dave

[davemee]

Hmm, I'm still struggling to get this to behave. I've create a new PR and tested on both iOS and Android (6.3.0), both of which should work the same way (so this should be testable and consistent on iOS and Android now)

I've been trying to work on the F^Ho^HRaccoon demo. They both render the video to a canvas using the CameraCapture plugin, but I can't get anything of Jeeliz to operate beyond this.

I get the following error on the Chrome debug console - this is a 'red' error, not a warning, so I suspect it's causing a bigger issue (however, I don't get this on iOS at all)

jeelizFaceTransfer.js:89 Uncaught ReferenceError: internalPixelFormat32f is not defined
    at d (jeelizFaceTransfer.js:89)
    at Object.i (jeelizFaceTransfer.js:90)
    at Object.i (jeelizFaceTransfer.js:43)
    at Object.V.init (jeelizFaceTransfer.js:144)
    at Object.init (ThreeJeelizHelper.js:253)
    at initWeboji (index.js:59)
    at Channel.onDeviceReady (index.js:48)
    at Channel.fire (cordova.js:847)
    at cordova.js:232

[xavierjs]

Hi,

I will take a look at this error.
Fortunately I managed to recover a macbook powerful enough to setup High Sierra and XCode10.
But this week I am traveling for work. I will test it on the Mac next week.

Best,
Xavier

[xavierjs]

Hi,

I have just pushed a fix regarding the error ReferenceError: internalPixelFormat32f is not defined.
I did not test using MacOSX yet but I will do it very soon.

Best,
Xavier

[davemee]

Hey Xavier,

Thanks for the updates! You know, this plugin also works on Android, so an Apple system isn't always needed - the broad process is the same on both platforms, but each may have different smaller errors.

In the meantime, I have error logs from both platforms - neither is picking up the video stream, I'm sad to say.

validateNewTexture:89: failed assertion `BytesPerRow of a buffer-backed texture with pixelFormat(MTLPixelFormatBGRA8Unorm) must be aligned to 64 bytes, found bytesPerRow(1200)'
(lldb)

(above: iOS Xcode console)

Uncaught ReferenceError: internalPixelFormat32f is not defined
    at d (jeelizFaceTransfer.js:89)
    at Object.i (jeelizFaceTransfer.js:90)
    at Object.i (jeelizFaceTransfer.js:43)
    at Object.V.init (jeelizFaceTransfer.js:144)
    at Object.init (ThreeJeelizHelper.js:253)
    at initWeboji (index.js:59)
    at Channel.onDeviceReady (index.js:48)
    at Channel.fire (cordova.js:847)
    at cordova.js:232
index.html:1 [.WebGL-0x9f119500]GL ERROR :GL_INVALID_OPERATION : glGenerateMipmap: Can not generate mips
index.html:1 [.WebGL-0x9f119500]RENDER WARNING: texture bound to texture unit 0 is not renderable. It might be non-power-of-2 or have incompatible texture filtering (maybe)?

(Above: Android, chromium web inspector console)

[xavierjs]

Hi @davemee

With Android I think that a webview component with the rights properties (to get camera access) can do the job to run Jeeliz libs.

But with IOS it is not possible and I am also interested to find a way to get video access. It is also very challenging :) .

The error abour internalPixelFormat32f it is very strange. I think your versions of jeelizFaceTransfer.js or jeelizFaceTransferES6.js were not updated since the last commit ( b1c63ed ). Indeed, the variable internalPixelFormat32f is not used anymore (I have just checked searching into the minified code to be sure).

I am currently trying stuffs to make it work with IOS.

Best,
Xavier

[xavierjs]

I managed to install the canvas test app on my Iphone XR with IOS 13.2 beta following these steps:

Updating MacOSX to High Sierra (10.14) and installing XCode 11 and brew.
Then I had to install some stuffs (it is maybe obvious, I am a Mac noob):

sudo npm install -g ionic
sudo npm install -g cordova
brew install ios-deploy
sudo xcode-select -s /Applications/Xcode.app/Contents/Developer

Then running from the /demos/cordova/canvas-camera-test:

rm package-lock.json
ionic cordova platform add ios@5.0.1
npm install
ionic cordova build ios

Then I had an error about the dev team. I opened the project using Xcode:

open platforms/ios/MyApp.xcodeproj

I changed bundle identifier to com.jeeliz.ios or whatever in General tab/Identity section
In General/Deployment info section, set Deployment target to 13.2 (or the right iPhone IOS version), Devices to iPhone

In Build Settings tab, go to Signing and set Code Signing Identity to IOS Developer

Plug Iphone, click on Product/Destination and set Device/Iphone
Click on Product/Build
Unlock the iphone and click on Product/Run

The icon of the App labeled with MyApp appeared on the phone.
When I clicked on I got an alert box as I am not allowed to open it.
On the phone I go to settings / general / Profiles & Device Management / Apple dev: ...
and I clicked on "Trust apple development: ...."

I went back, I clicked on the app again and I got the video working :)

Now I have to add the weboji lib :)

[xavierjs]

Wow I just discovered that we can access to the web console as if it were a web app using https://geeklearning.io/apache-cordova-and-remote-debugging-on-ios/

[xavierjs]

I have found the issue :)
In fact the neural network model was never loaded, so the drawing loop never started.
The neural network was loaded with an AJAX request.

With Cordova/Safari there is a bug: xmlHttp.status is always set to 0 when xmlHttp.readyState === 4 even if the answer is not empty (it should be set to 200.

I will add a workaround and push very soon.

[xavierjs]

@davemee I updated the iOS cordova raccoon demo.
It works perfectly on my Iphone XR.

Best,
Xavier

[davemee]

Confimed - works beautifully. Thanks for your perseverance, Xavier, and great suite of code!

[xavierjs]

Thanx a lot for you help @davemee . We are sometimes asked to make the library work also on a native environment. The typical use case is a customer who want a native app, but who also have an older native one and who want to be able to only develop 1 solution for the 2 use cases.

[xavierjs]

BREAKING NEWS: Camera access is enabled on IOS14.3 which is released this Monday ( December 14th ). So Weboji will work on Chrome, Firefox, Opera for IOS, and into WKWebviews embedded in native apps.