Support for simulcast in Android SDK #3

davidzhao · 2021-09-10T21:49:50Z

Adapted from Shiguredo's patch

https://github.com/shiguredo-webrtc-build/webrtc-build/blob/master/patches/android_simulcast.patch

Adapted from Shiguredo's patch https://github.com/shiguredo-webrtc-build/webrtc-build/blob/master/patches/android_simulcast.patch

cloudwebrtc

LGTM

cloudwebrtc · 2021-10-05T14:49:41Z

I created a PR here #9.

davidzhao · 2021-10-05T17:30:10Z

@voluntas we very much appreciate your efforts to improve WebRTC on mobile, and making them available for others! As @cloudwebrtc mentioned, we will attribute all changes to their original sources. If you have any other suggestions regarding attributions or license compliance, please let us know.

Schedule the frames to be decoded based on the pacing delay from the last decode scheduled time. In the current implementation, multiple threads and different functions in same thread can call MaxWaitingTime(), thereby increasing the wait time each time the function is called. Instead of returning the wait time for a future frame based on the number of times the function is called, return the wait time only for the next frame to be decoded. Threads can call the function repeatedly to check the waiting time for next frame and wake up and then go back to waiting if an encoded frame is not available. (cherry picked from commit 82c2248) Change-Id: I00886c1619599f94bde5d5eb87405572e435bd73 Bug: chromium:1237402 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/226502 Reviewed-by: Johannes Kron <kron@webrtc.org> Reviewed-by: Philip Eliasson <philipel@webrtc.org> Commit-Queue: Johannes Kron <kron@webrtc.org> Cr-Original-Commit-Position: refs/heads/master@{#34660} Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/228532 Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Cr-Commit-Position: refs/branch-heads/4577@{#3} Cr-Branched-From: 5196931-refs/heads/master@{#34463}

This reverts commit dbab1be. Reason for revert: Breaks VP9 media performance under heavy packet loss. Original change's description: > Always unwrap VP9 TL0PicIdx forward if the frame is newer. > > Bug: webrtc:12979, chromium:1245564 > Change-Id: Idcc14f8f61b04f9eb194b55ffa40fb95319a881c > Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/226463 > Reviewed-by: Danil Chapovalov <danilchap@webrtc.org> > Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> > Reviewed-by: Mirko Bonadei <mbonadei@webrtc.org> > Commit-Queue: Philip Eliasson <philipel@webrtc.org> > Cr-Commit-Position: refs/heads/master@{#34513} # Not skipping CQ checks because original CL landed > 1 day ago. (cherry picked from commit 0d17535) Bug: webrtc:12979 Change-Id: Id315db8d67143372724448b8801a86aee9a2f0aa Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/230422 Reviewed-by: Philip Eliasson <philipel@webrtc.org> Reviewed-by: Danil Chapovalov <danilchap@webrtc.org> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Commit-Queue: Philip Eliasson <philipel@webrtc.org> Cr-Original-Commit-Position: refs/heads/main@{#34863} Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/231133 Cr-Commit-Position: refs/branch-heads/4606@{#3} Cr-Branched-From: 8b18304-refs/heads/master@{#34737}

this could happen if setCodecPreferences is used to prefer red over opus as it is done for red+opus. BUG=webrtc:13287,chromium:1271135 (cherry picked from commit 60c01cc) No-Try: True Change-Id: I3d61cd8f1a364572bc531a75dcc239c3919138cc Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/237800 Reviewed-by: Harald Alvestrand <hta@webrtc.org> Commit-Queue: Philipp Hancke <philipp.hancke@googlemail.com> Cr-Original-Commit-Position: refs/heads/main@{#35344} Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/239440 Cr-Commit-Position: refs/branch-heads/4692@{webrtc-sdk#3} Cr-Branched-From: c276aee-refs/heads/main@{#35313}

Support for simulcast in Android SDK

It's normal for a receiver to not be configured to receive, such as when currentDirection is not (or not yet) "sendrecv" or "recvonly". getParameters() returning an empty set of encodings is valid and these logs are not very useful. It's also inconsistent that we only log after SLD has happened due to different code paths inside getParameters(), repro: https://jsfiddle.net/henbos/xqksj3wd/. Most notably we're calling getParameters() internally from inside of getStats() which can cause excessive log spam. I prefer that we remove these logs rather than avoid calling getParameters() from inside of getStats() on non-receiving receivers since it's valid to check how many encodings exist on a receiver using getParameters(), and whether or not the SSRC has been signaled could in theory affect the number of encodings even if we do want to receive. Also an app calling getParameters() on an inactive receiver is valid and should not cause logs. (cherry picked from commit 73ee252) No-Try: True Bug: webrtc:14225, chromium:1339762 Change-Id: I4290781d6aed92aa03fe0c662762aa97c99a045c Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/266960 Commit-Queue: Erik Språng <sprang@webrtc.org> Auto-Submit: Henrik Boström <hbos@webrtc.org> Reviewed-by: Erik Språng <sprang@webrtc.org> Commit-Queue: Henrik Boström <hbos@webrtc.org> Cr-Original-Commit-Position: refs/heads/main@{#37335} Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/267060 Commit-Queue: Mirko Bonadei <mbonadei@webrtc.org> Cr-Commit-Position: refs/branch-heads/5112@{#3} Cr-Branched-From: a976a87-refs/heads/main@{#37168}

Support for simulcast in Android SDK

…n) P2PTransportChannel can then use either of the ICE controller factories configured with field trials. Bug: webrtc:14367, webrtc:14131 Change-Id: I09ab99673d6ef81f56abe88987f5b67d84c24cb5 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/271292 Reviewed-by: Jonas Oreland <jonaso@webrtc.org> Reviewed-by: Per Kjellander <perkj@webrtc.org> Commit-Queue: Sameer Vijaykar <samvi@google.com> Cr-Commit-Position: refs/heads/main@{#38076}

Support for simulcast in Android SDK

… pacer queue. This metric was always supposed to be the spec's answer to googBucketDelay, and is defined as "The total number of seconds that packets have spent buffered locally before being transmitted onto the network." But our implementation measured the time between capture and send, including encode time. This is incorrect and yields a much larger value than expected. This CL updated the metric to do what the spec says. Implementation-wise we measure the time between pushing and popping each packet from the queue (in modules/pacing/prioritized_packet_queue.cc). The spec says to increment the delay counter at the same time as we increment the packet counter in order for the app to be able to do "delta totalPacketSendDelay / delta packetSent". For this reason, `total_packet_delay` is added to RtpPacketCounter. (Previously, the two counters were incremented on different threads and observers.) Running Google Meet on a good network, I could observe a 2-3 ms average send delay per packet with this implementation compared to 20-30 ms with the old implementation. See b/137014977#comment170 for comparison with googBucketDelay which is a little bit different by design - totalPacketSendDelay is clearly better than googBucketDelay. Since none of this depend on the media kind, we can wire up this metric for audio as well in a follow-up: https://webrtc-review.googlesource.com/c/src/+/280523 (cherry picked from commit d819921) Bug: webrtc:14593, chromium:1378895 Change-Id: If8fcd82fee74030d0923ee5df2c2aea2264600d4 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/280443 Reviewed-by: Erik Språng <sprang@webrtc.org> Commit-Queue: Henrik Boström <hbos@webrtc.org> Cr-Original-Commit-Position: refs/heads/main@{#38480} Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/281160 Reviewed-by: Harald Alvestrand <hta@webrtc.org> Cr-Commit-Position: refs/branch-heads/5359@{webrtc-sdk#3} Cr-Branched-From: fb3bd4a-refs/heads/main@{#38387}

rtc::CopyOnWriteBuffer::SetSize extends buffer with uninitialized memory by design. It is up to the user of the rtc::CopyOnWriteBuffer to ensure it is initialized. (cherry picked from commit f52e015) No-Try: true Bug: chromium:1403397 Change-Id: Ic0111a84bda32379770ddb1c7d24bee10d96b7a4 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/289041 Reviewed-by: Rasmus Brandt <brandtr@webrtc.org> Commit-Queue: Danil Chapovalov <danilchap@webrtc.org> Cr-Original-Commit-Position: refs/heads/main@{#38959} Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/291540 Reviewed-by: Per Kjellander <perkj@webrtc.org> Cr-Commit-Position: refs/branch-heads/5481@{#3} Cr-Branched-From: 2e1a9a4-refs/heads/main@{#38901}

This CL migrates unit tests to the new TaskQueueBase interface. Bug: chromium:1416199 Change-Id: Ic15c694b28eb67450ac99fdd56754de1246a4d95 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/295621 Commit-Queue: Markus Handell <handellm@webrtc.org> Reviewed-by: Danil Chapovalov <danilchap@webrtc.org> Reviewed-by: Harald Alvestrand <hta@webrtc.org> Cr-Commit-Position: refs/heads/main@{#39434}

This CL migrates the task queue paced sender unit test to the new TaskQueueBase interface. Bug: chromium:1416199 Change-Id: Id0568bb9a08bf43b92e33fdf45fe75a57e5a7a27 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/295722 Reviewed-by: Harald Alvestrand <hta@webrtc.org> Commit-Queue: Markus Handell <handellm@webrtc.org> Cr-Commit-Position: refs/heads/main@{#39436}

This CL completes migration to the new TaskQueueBase interface permitting location tracing in Chrome. Bug: chromium:1416199 Change-Id: Iff7ff5796752a1520384a3db0135a1d4b9438988 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/294540 Reviewed-by: Harald Alvestrand <hta@webrtc.org> Commit-Queue: Markus Handell <handellm@webrtc.org> Reviewed-by: Danil Chapovalov <danilchap@webrtc.org> Cr-Commit-Position: refs/heads/main@{#39439}

This CL forwards repeating task client locations to the passed task queue. Bug: chromium:1416199 Change-Id: I437d596f8d327d13498b47dfc0a03812af870331 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/295623 Reviewed-by: Harald Alvestrand <hta@webrtc.org> Commit-Queue: Markus Handell <handellm@webrtc.org> Cr-Commit-Position: refs/heads/main@{#39443}

This CL forwards TaskQueue locations to the contained task queue. Bug: chromium:1416199 Change-Id: I989ae445a67991bf5a857407135dbe8bacbd3c55 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/295622 Reviewed-by: Harald Alvestrand <hta@webrtc.org> Commit-Queue: Markus Handell <handellm@webrtc.org> Cr-Commit-Position: refs/heads/main@{#39446}

Support for simulcast in Android SDK

Simulcast support for iOS SDK (#4) Support for simulcast in Android SDK (#3) include simulcast headers for mac also (#10) Fix simulcast using hardware encoder on Android (#48) Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> Co-authored-by: Angelika Serwa <angelika.serwa@gmail.com>

This is required for ReportTransportStats since iterating over the transceiver list from the network thread is not safe. (cherry picked from commit dba22d3) No-Try: true Bug: chromium:1446274, webrtc:12692 Change-Id: I7c514df9f029112c4b1da85826af91217850fb26 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/307340 Reviewed-by: Harald Alvestrand <hta@webrtc.org> Commit-Queue: Tomas Gunnarsson <tommi@webrtc.org> Cr-Original-Commit-Position: refs/heads/main@{#40197} Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/308001 Reviewed-by: Mirko Bonadei <mbonadei@webrtc.org> Cr-Commit-Position: refs/branch-heads/5735@{#3} Cr-Branched-From: df7df19-refs/heads/main@{#39949}

Simulcast support for iOS SDK (#4) Support for simulcast in Android SDK (#3) include simulcast headers for mac also (#10) Fix simulcast using hardware encoder on Android (#48) Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> Co-authored-by: Angelika Serwa <angelika.serwa@gmail.com>

This is required for ReportTransportStats since iterating over the transceiver list from the network thread is not safe. (cherry picked from commit dba22d3) No-Try: true Bug: chromium:1446274, webrtc:12692 Change-Id: I7c514df9f029112c4b1da85826af91217850fb26 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/307340 Reviewed-by: Harald Alvestrand <hta@webrtc.org> Commit-Queue: Tomas Gunnarsson <tommi@webrtc.org> Cr-Original-Commit-Position: refs/heads/main@{#40197} Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/308001 Reviewed-by: Mirko Bonadei <mbonadei@webrtc.org> Cr-Commit-Position: refs/branch-heads/5735@{#3} Cr-Branched-From: df7df19-refs/heads/main@{#39949}

Simulcast support for iOS SDK (#4) Support for simulcast in Android SDK (#3) include simulcast headers for mac also (#10) Fix simulcast using hardware encoder on Android (#48) Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> Co-authored-by: Angelika Serwa <angelika.serwa@gmail.com>

Simulcast support for iOS SDK (#4) Support for simulcast in Android SDK (#3) include simulcast headers for mac also (#10) Fix simulcast using hardware encoder on Android (#48) Add scalabilityMode support for AV1/VP9. (#90) Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> Co-authored-by: Angelika Serwa <angelika.serwa@gmail.com>

Use M125 as the latest version and migrate historical patches to m125 Patches Group: ## 1. Update README.md b6c65fc * Add Apache-2.0 license and some note to README.md. (#9) * Updated readme detailing changes from original (#42) * Adding membrane framework (#51) * Updated readme (#83) ## 2. Audio Device Optimization 7454824 * allow listen-only mode in AudioUnit, adjust when category changes (#2) * release mic when category changes (#5) * Change defaults to iOS defaults (#7) * Sync audio session config (#8) * feat: support bypass voice processing for iOS. (#15) * Remove MacBookPro audio pan right code (#22) * fix: Fix can't open mic alone when built-in AEC is enabled. (#29) * feat: add audio device changes detect for windows. (#41) * fix Linux compile (#47) * AudioUnit: Don't rely on category switch for mic indicator to turn off (#52) * Stop recording on mute (turn off mic indicator) (#55) * Cherry pick audio selection from m97 release (#35) * [Mac] Allow audio device selection (#21) * RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (#80) * Allow custom audio processing by exposing AudioProcessingModule (#85) * Expose audio sample buffers for Android (#89) * feat: add external audio processor for android. (#103) * android: make audio output attributes modifiable (#118) * Fix external audio processor sample rate calculation (#108) * Expose remote audio sample buffers on RTCAudioTrack (#84) * Fix memory leak when creating audio CMSampleBuffer #86 ## 3. Simulcast/SVC support for iOS/Android. b0b9fe9 - Simulcast support for iOS SDK (#4) - Support for simulcast in Android SDK (#3) - include simulcast headers for mac also (#10) - Fix simulcast using hardware encoder on Android (#48) - Add scalabilityMode support for AV1/VP9. (#90) ## 4. Android improvements. 9aaaab5 - Start/Stop receiving stream method for VideoTrack (#25) - Properly remove observer upon deconstruction (#26) - feat: Expose setCodecPreferences/getCapabilities for android. (#61) - fix: add WrappedVideoDecoderFactory.java. (#74) ## 5. Darwin improvements a13ea17 - [Mac/iOS] feat: Add RTCYUVHelper for darwin. (#28) - Cross-platform `RTCMTLVideoView` for both iOS / macOS (#40) - rotationOverride should not be assign (#44) - [ObjC] Expose properties / methods required for AV1 codec support (#60) - Workaround: Render PixelBuffer in RTCMTLVideoView (#58) - Improve iOS/macOS H264 encoder (#70) - fix: fix video encoder not resuming correctly upon foregrounding (#75). - add PrivacyInfo.xcprivacy to darwin frameworks. (#112) - Add NSPrivacyCollectedDataTypes key to xcprivacy file (#114) - Thread-safe `RTCInitFieldTrialDictionary` (#116) - Set RTCCameraVideoCapturer initial zoom factor (#121) - Unlock configuration before starting capture session (#122) ## 6. Desktop Capture for macOS. 841d78f - [Mac] feat: Support screen capture for macOS. (#24) (#36) - fix: Get thumbnails asynchronously. (#37) - fix: Use CVPixelBuffer to build DesktopCapture Frame, fix the crash caused by non-CVPixelBuffer frame in RTCVideoEncoderH264 that cannot be cropped. (#63) - Fix the crash when setting the fps of the virtual camera. (#62) ## 7. Frame Cryptor Support. fc08745 - feat: Frame Cryptor (aes gcm/cbc). (#54) - feat: key ratchet/derive. (#66) - fix: skip invalid key when decryption failed. (#81) - Improve e2ee, add setSharedKey to KeyProvider. (#88) - add failure tolerance for framecryptor. (#91) - fix h264 freeze. (#93) - Fix/send frame cryptor events from signaling thread (#95) - more improvements for E2EE. (#96) - remove too verbose logs (#107) - Add key ring size to keyProviderOptions. (#109) ## 8. Other improvements. eed6c8a - Added yuv_helper (#57) - ABGRToI420, ARGBToI420 & ARGBToRGB24 (#65) - more yuv wrappers (#87) - Fix naming for yuv helper (#113) - Fix missing `RTC_OBJC_TYPE` macros (#100) --------- Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> Co-authored-by: David Zhao <dz@livekit.io> Co-authored-by: davidliu <davidliu@deviange.net> Co-authored-by: Angelika Serwa <angelika.serwa@gmail.com> Co-authored-by: Théo Monnom <theo.monnom@outlook.com>

Simulcast support for iOS SDK (webrtc-sdk#4) Support for simulcast in Android SDK (webrtc-sdk#3) include simulcast headers for mac also (webrtc-sdk#10) Fix simulcast using hardware encoder on Android (webrtc-sdk#48) Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> Co-authored-by: Angelika Serwa <angelika.serwa@gmail.com>

Support for simulcast in Android SDK

aeee3a0

Adapted from Shiguredo's patch https://github.com/shiguredo-webrtc-build/webrtc-build/blob/master/patches/android_simulcast.patch

davidzhao requested a review from cloudwebrtc September 10, 2021 21:50

cloudwebrtc approved these changes Sep 11, 2021

View reviewed changes

cloudwebrtc merged commit dfec53e into main Sep 11, 2021

davidzhao deleted the android-simulcast branch September 11, 2021 18:39

davidliu pushed a commit to davidliu/webrtc that referenced this pull request Feb 8, 2022

Merge pull request webrtc-sdk#3 from webrtc-sdk/android-simulcast

c7b45e5

Support for simulcast in Android SDK

davidliu pushed a commit to davidliu/webrtc that referenced this pull request Feb 9, 2022

Merge pull request webrtc-sdk#3 from webrtc-sdk/android-simulcast

febc492

Support for simulcast in Android SDK

davidliu pushed a commit that referenced this pull request Feb 10, 2022

Merge pull request #3 from webrtc-sdk/android-simulcast

1388778

Support for simulcast in Android SDK

davidliu pushed a commit that referenced this pull request Jul 17, 2022

Merge pull request #3 from webrtc-sdk/android-simulcast

d27e2ce

Support for simulcast in Android SDK

davidliu pushed a commit that referenced this pull request Jul 21, 2022

Merge pull request #3 from webrtc-sdk/android-simulcast

41d8773

Support for simulcast in Android SDK

vickyleu referenced this pull request Jul 28, 2022

fix issue for screen sharing.

e958e66

cloudwebrtc added a commit that referenced this pull request Jan 18, 2023

Merge pull request #3 from webrtc-sdk/android-simulcast

f833af6

Support for simulcast in Android SDK

cloudwebrtc added a commit that referenced this pull request Jun 5, 2023

Merge pull request #3 from webrtc-sdk/android-simulcast

d2da856

Support for simulcast in Android SDK

cloudwebrtc mentioned this pull request Jun 12, 2023

update to m114 #76

Merged

cloudwebrtc mentioned this pull request May 21, 2024

Update to m125. #119

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for simulcast in Android SDK #3

Support for simulcast in Android SDK #3

davidzhao commented Sep 10, 2021

cloudwebrtc left a comment

cloudwebrtc commented Oct 5, 2021

davidzhao commented Oct 5, 2021

Support for simulcast in Android SDK #3

Support for simulcast in Android SDK #3

Conversation

davidzhao commented Sep 10, 2021

cloudwebrtc left a comment

Choose a reason for hiding this comment

cloudwebrtc commented Oct 5, 2021

davidzhao commented Oct 5, 2021