iOS 14.3 WKWebView

[manit77]

We've had WebRTC video calls running successfully for years on Chrome, Android, iOS and FireFox. I am testing to see if Twilio can replace some parts of our system. We are testing the twilio-video.js on iOS 14.3. The call process works as expected using Chrome, Safari. When embedding our web page inside WKWebView and we ran into some issues using Twilio. I know WKWebView is not listed as a supported browser but has anyone tested video calls in WKWebView?

When using twilio-video.js:

• No JavaScript errors are reported.

• Local camera and audio works using getUserMedia.

• The local participant can connect to a room but no video or audio streams are sent to Twilio.

• No remote audio or video streams are received from Twilio.

• It appears that these events never gets triggered:
  participant.on('trackSubscribed', track => trackSubscribed(track));
  participant.on('trackUnsubscribed', trackUnsubscribed);

• When removing twilio-video.js and using our own JavaScript WebRTC call process inside WKWebView. Everything works as expected.

• When loading a third party web page such as https://janus.conf.meetecho.com/videocalltest.html inside WKWebView. Everything works as expected.

Any feedback is appreciated. Thanks.

Update:

I did find oner error during the negotiation process.

Calling setRemoteDescription with an RTCSessionDescription of type "answer" failed with the error "Failed to set remote answer sdp: The order of m-lines in answer doesn't match order in offer. Rejecting answer.".

[ma9o]

+1

[manjeshbhargav]

Hi @manit77 ,

Thanks for reporting this issue. We don't have WKWebView support in our roadmap for 2021 as of now. However, I'm curious as to why WKWebView is not working, since it is essentially a Safari window. Can you share the JavaScript console logs from your test app if possible? You can enable debug logging in the SDK by setting logLevel: 'debug' in ConnectOptions.

[manit77]

twilio-log1.txt

It is odd that your library works perfectly on Safari and on our system works just fine inside WKWebView. We setup small-room and recording in Twilio and our system uses Peer to Peer with turn servers enforced.

Here is the file attached. This is in our log format but I should have captured all the values. I see the error around line 127. I tested this on iPhone 7 14.3, iPad Mini 14.4 with the same results.

Thanks for looking at it.

[manit77]

inside the log file, U is peer 1 and I is peer 2

[greg-powerfront]

We also have a big need for this and a lot of clients wanting to take client phone calls in the app using WKWebView.
We are facing the same issue and it looks like it might be a minor fix on Twilio side to get this functional.
Our clients are already using Video Chat with the twilio integration on normal website browsers and we prefer not have to go with another provider to get this working in our App SDK with works in a WKWebView.

[LuisaMontanez]

+1

[manit77]

@powerfront,

Are you using small-room or peer to peer in your Twilio calls?

[noahmehl]

@manjeshbhargav Surprisingly, WKWebView is quite different than Safari in iOS. In fact, Apple only added some WebRTC functionality in iOS 14.3, getUserMedia(): https://webkit.org/blog/11353/mediarecorder-api/

We've (Open Telecom Foundation) been working on: https://github.com/OpenTelecom/WKWebViewRTC and https://cocoapods.org/pods/WKWebViewRTC. We've recently updated for getUserMedia() for audio and video (for both 14.3 and previous to that).

Looks like the Twilio example doesn't quite work fully for WKWebViewRTC either. We'll look into it.

[arturfps]

+1

[greg-powerfront]

@powerfront,

Are you using small-room or peer to peer in your Twilio calls?

Peer to peer

Thks

[LiuGuanwei77]

+1

[cbxp]

@manjeshbhargav
We have submitted a pull request to solve the issue: twilio/twilio-webrtc.js#133

The problem is that a function in twilio-webrtc.js project which guesses the browser by user agent does not work correctly when a web application is running inside WKWebView.

[manit77]

@cbxp

Also need to add support for iPad. this the user agent for iPad mini. I can't copy paste it, this is typed:

Mozilla/5.0 (iPad; CPU OS 14_4 like Mac OS X) AppleWebkit/605.1.15 (KHTML, like Gecko) Mobile/15E148

[manit77]

@cbxp

I tested your fix on the iPad. It is working. I added iPad to the user agent check.

if (/Safari|iPhone|iPad/.test(userAgent)) {

Thanks!

[cbxp]

Thanks @manit77
Pull request twilio/twilio-webrtc.js#133 includes iPad fix as well

@manjeshbhargav - could you please review and accept the pull request?

[aponski]

+1

[jnorris441]

+1

[sowsan]

Any idea when this fix will be available in master and ready to use in production?

[tbasallo]

I can confirm that this fixes the issue. I modified the CDN file to include "iPhone" in the browser test and served locally and it worked like a charm.

Can someone from the Twilio team provide some feedback?

(you have to love open source - these moments are great.)

[manjeshbhargav]

Hi everyone,

We have WKWebView support tentatively as part of our Q2 roadmap, and we will actively work on it soon. @cbxp , thanks for the PR. We will certainly incorporate it when we start working on this.

Thanks,

Manjesh

[sowsan]

This is a major blocker for one of my projects. What is the best way to get a fix temporarily? We are even thinking of switching to other Video SDK providers if this is going to take more time..

[renato-bohler]

This is a major blocker for one of my projects. What is the best way to get a fix temporarily? We are even thinking of switching to other Video SDK providers if this is going to take more time..

I've successfully used patch-package to use @cbxp's fix on the twilio-webrtc package. You could try this while the fix is not officially available.

[sowsan]

This is a major blocker for one of my projects. What is the best way to get a fix temporarily? We are even thinking of switching to other Video SDK providers if this is going to take more time..

I've successfully used patch-package to use @cbxp's fix on the twilio-webrtc package. You could try this while the fix is not officially available.

That is great and thanks for your prompt response. I have not used patch package before. Can you share the steps to use it pls?

[renato-bohler]

This is a major blocker for one of my projects. What is the best way to get a fix temporarily? We are even thinking of switching to other Video SDK providers if this is going to take more time..

I've successfully used patch-package to use @cbxp's fix on the twilio-webrtc package. You could try this while the fix is not officially available.

That is great and thanks for your prompt response. I have not used patch package before. Can you share the steps to use it pls?

Sure! You just need to follow the instructions on their readme. But, to simplify:

1. Install patch-package: npm install patch-package --save-dev or yarn install -D patch-package
2. In your node_modules, search for the lib/util/index.js file inside the @twilio/webrtc folder
3. Open the file and make the same changes that @cbxp has done on the file in his PR
4. Run patch-package: npx run patch-package @twilio/webrtc or yarn patch-package @twilio/webrtc. This will create a folder called patches which will include a diff that patches that dependency.
5. Add the following script to your package.json. This will make patch-package apply the patches that are on your repo's patches folder into your node_modules every time you (or anyone else) install dependencies.

 "scripts": {
+  "postinstall": "patch-package"
 }

Note: I haven't tested this step-by-step, but that should be it as far as I remember.

[mosaabramadan]

i confirm the issue has been fixed with the PR provided by @cbxp

I simply replaced the index.js file in @twilio/webrtc/lib/util and it all is well across all platforms

[hthetiot]

Browser userAgent Snifing is bad in 2021 use features detection.

[it4e]

This is a major blocker for one of my projects. What is the best way to get a fix temporarily? We are even thinking of switching to other Video SDK providers if this is going to take more time..

Another solution is to set the user-agent of WKWebView to contain the word "Safari", then we don't have to worry about making any changes to the Twilio code.

[fbruckhoff]

Apple and the WebKit team both confirmed to me that WKWebView currently doesn't support accessing the microphone or camera in the background. So while you can hear other people talk, as soon as you background the app / lock the screen, others can no longer hear or see you.

Were you able to solve this in twilio-webrtc or WKWebViewRTC? cc @noahmehl

[noahmehl]

@fbruckhoff this has not been tested for WKWebViewRTC as far as I know. All of the WKWebViewRTC is based on the work by @hthetiot for the Cordova project: https://github.com/cordova-rtc/cordova-plugin-iosrtc, and I'm sure he'd have a better idea what is possible here.

[fbruckhoff]

Thanks @noahmehl! @hthetiot it'd be great to hear whether / how you were able to get around the background mic issue :) Cheers

[hthetiot]

For using audio and microphone while the application is minimized in background we recommend using cordova-plugin-background-mode in combination with cordova-plugin-iosrtc see sample application here: https://github.com/cordova-rtc/cordova-plugin-iosrtc-sample

Using background Cordova plugin results in a red status bar with microphone in use and you can still hear remote peer audio when the application is in background. However video will not be sent to remote peer while in background.

See: https://github.com/katzer/cordova-plugin-background-mode

[oanguenot]

Hi all,
I have a similar issue with a WKWebView component in a Mac desktop app. I used the same workaround proposed (by changing the user-agent) and it's work fine now.

Could someone confirm that this issue will address WKWebVIew whatever the platform ?

Here is a trace of the issue encountered on my own:
"Calling setRemoteDescription with an RTCSessionDescription of type "answer" failed with the error "Failed to set remote answer sdp: The order of m-lines in answer doesn't match order in offer. Rejecting answer."
TwilioError: Client is unable to apply a remote media description

If this is not the same issue, I can create a new one. Thanks in avance for your answer

[noahmehl]

@hthetiot Thanks for that!!

[hthetiot]

@oanguenot yes following issue is most likely related, since when safari (via user agent) is detected to my knowledge the twilio library fix the order or m-lines. In any case if you don't have the issue after applying the user agent fix not need to create new issue in my POV.

Here is a trace of the issue encountered on my own:
"Calling setRemoteDescription with an RTCSessionDescription of type "answer" failed with the error "Failed to set remote answer sdp: The order of m-lines in answer doesn't match order in offer. Rejecting answer."
TwilioError: Client is unable to apply a remote media description

[oanguenot]

@hthetiot Thanks you for your answer

[PikaJoyce]

Hi Folks,

The latest SDK release 2.21.0 now supports WebViews on iOS.

Thanks for all of the patience! If any issues arise, please don't hesitate to open another issue with us. With that being said, I will now be closing this issue.

Best,
Joyce