Improve H264 encoder #70

hiroshihorie · 2023-05-18T08:26:31Z

Improve bitrate
Turn on Apple's optimizations when available
maxQP for screen share mode
Some code clean up

davidzhao

looks good! (for the parts I was able to understand). I'm still not quite sure what the bug was that prevented it from maintaining bitrate previously.

davidzhao · 2023-05-22T00:43:05Z

sdk/objc/components/video_codec/RTCVideoEncoderH264.mm

-  _encoderFrameRate = MIN(settings.maxFramerate, _maxAllowedFrameRate);
+  if (_encodeMode == Constant) {
+    _targetBitrateBps = _maxBitrate;
+    // _bitrateAdjuster->SetTargetBitrateBps(_targetBitrateBps);


nit: this can be removed?

davidzhao · 2023-05-22T00:44:46Z

sdk/objc/components/video_codec/RTCVideoEncoderH264.mm


+  CMTime presentationTimeStamp = CMTimeMake(frame.timeStampNs / rtc::kNumNanosecsPerMillisec, 1000);
+  if (CMTimeCompare(presentationTimeStamp, _previousPresentationTimeStamp) == 0) {


when does this happen? Is it an optimization for screenshare?

This is extra safety to avoid this issue:
https://developer.apple.com/forums/thread/702891

thanks! looks like this is part of reason why it's unable to keep up.

davidzhao · 2023-05-22T01:17:49Z

sdk/objc/components/video_codec/RTCVideoEncoderH264.mm

@@ -811,7 +910,12 @@ - (void)frameWasEncoded:(OSStatus)status
    RTC_LOG(LS_ERROR) << "Encode callback failed";
    return;
  }
-  _bitrateAdjuster->Update(frame.buffer.length);
+
+  // if (_encodeMode == Constant) {


is this because we've deprecated bitrateAdjuster? good to remove then?

I tested two days ago only modifying kLimitToAverageBitRateFactor = 1.5f; to
kLimitToAverageBitRateFactor = 5.0f; also seems to get good results (keep bitrateAdjuster). will do a comparison test soon

@hiroshihorie , can you try just modifying kLimitToAverageBitRateFactor = 5.0f based on the original code?

@davidzhao Yes I think it's ok to remove _bitrateAdjuster.
@cloudwebrtc kLimitToAverageBitRateFactor 5~8 seems fine but, I think chrome was using 10.0f if my memory was correct. Should we just follow what chrome uses ?

@cloudwebrtc kLimitToAverageBitRateFactor 5~8 seems fine but, I think chrome was using 10.0f if my memory was correct. Should we just follow what chrome uses ?

following the chrome should be fine, and test results are as expected, so great work @hiroshihorie

davidzhao · 2023-05-22T01:19:03Z

sdk/objc/components/video_codec/RTCVideoEncoderH264.mm

@@ -54,14 +54,42 @@ - (void)frameWasEncoded : (OSStatus)status flags : (VTEncodeInfoFlags)infoFlags
 // The ratio between kVTCompressionPropertyKey_DataRateLimits and
 // kVTCompressionPropertyKey_AverageBitRate. The data rate limit is set higher
 // than the average bit rate to avoid undershooting the target.
-const float kLimitToAverageBitRateFactor = 1.5f;
+const float kLimitToAverageBitRateFactor = 10.0f;


any idea why this value had to be set so high compared to what it was before? it seems that 10x average bitrate would seem quite high, no?

Just did some tests for camera capture, and the bit rate will be close to the target value (but not exceeding the limit). the moving picture is smooth and clear, and the bit rate of the static picture can be reduced to 1/5 of the target bit rate

The game screen may be due to complex textures, at least 4~6mbps needs to be set to achieve smooth and clear 720p/30fps, and the camera only needs 2mbps, so this PR is good for me

@davidzhao This is for computing data rate limits and not the average bitrate.

got it, I don't really understand this so feel free to ignore :)

cloudwebrtc

lgtm

hiroshihorie · 2023-05-22T04:54:48Z

Chrome has both CBR mode and VBR(ABR) mode, not sure in what condition that switches. Will take a quick look.

sdk/objc/native/src/objc_video_track_source.mm

* progress * enable low-latency video encoding * fix compile * prioritize speed * fix required os versions * maxQP for screensharing * cbr * Update RTCVideoEncoderH264.mm * Update RTCVideoEncoderH264.mm * format * clean up * resolution alignment * revert resolution alignment

Added yuv_helper (#57) ABGRToI420, ARGBToI420 & ARGBToRGB24 (#65) Improve iOS/macOS H264 encoder (#70) Co-authored-by: Théo Monnom <theo.monnom@outlook.com> Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com>

[Mac/iOS] feat: Add RTCYUVHelper for darwin. (#28) Cross-platform `RTCMTLVideoView` for both iOS / macOS (#40) rotationOverride should not be assign (#44) [ObjC] Expose properties / methods required for AV1 codec support (#60) Workaround: Render PixelBuffer in RTCMTLVideoView (#58) Improve iOS/macOS H264 encoder (#70) Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com>

[Mac/iOS] feat: Add RTCYUVHelper for darwin. (#28) Cross-platform `RTCMTLVideoView` for both iOS / macOS (#40) rotationOverride should not be assign (#44) [ObjC] Expose properties / methods required for AV1 codec support (#60) Workaround: Render PixelBuffer in RTCMTLVideoView (#58) Improve iOS/macOS H264 encoder (#70) fix: fix video encoder not resuming correctly upon foregrounding (#75). Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com>

[Mac/iOS] feat: Add RTCYUVHelper for darwin. (#28) Cross-platform `RTCMTLVideoView` for both iOS / macOS (#40) rotationOverride should not be assign (#44) [ObjC] Expose properties / methods required for AV1 codec support (#60) Workaround: Render PixelBuffer in RTCMTLVideoView (#58) Improve iOS/macOS H264 encoder (#70) fix: fix video encoder not resuming correctly upon foregrounding (#75). add PrivacyInfo.xcprivacy to darwin frameworks. (#112) Add NSPrivacyCollectedDataTypes key to xcprivacy file (#114) Thread-safe `RTCInitFieldTrialDictionary` (#116) Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com>

[Mac/iOS] feat: Add RTCYUVHelper for darwin. (#28) Cross-platform `RTCMTLVideoView` for both iOS / macOS (#40) rotationOverride should not be assign (#44) [ObjC] Expose properties / methods required for AV1 codec support (#60) Workaround: Render PixelBuffer in RTCMTLVideoView (#58) Improve iOS/macOS H264 encoder (#70) fix: fix video encoder not resuming correctly upon foregrounding (#75). Fix camera rotation (#92) add PrivacyInfo.xcprivacy to darwin frameworks. (#112) Add NSPrivacyCollectedDataTypes key to xcprivacy file (#114) Thread-safe `RTCInitFieldTrialDictionary` (#116) Set RTCCameraVideoCapturer initial zoom factor (#121) Unlock configuration before starting capture session #122 Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com>

Use M125 as the latest version and migrate historical patches to m125 Patches Group: ## 1. Update README.md b6c65fc * Add Apache-2.0 license and some note to README.md. (#9) * Updated readme detailing changes from original (#42) * Adding membrane framework (#51) * Updated readme (#83) ## 2. Audio Device Optimization 7454824 * allow listen-only mode in AudioUnit, adjust when category changes (#2) * release mic when category changes (#5) * Change defaults to iOS defaults (#7) * Sync audio session config (#8) * feat: support bypass voice processing for iOS. (#15) * Remove MacBookPro audio pan right code (#22) * fix: Fix can't open mic alone when built-in AEC is enabled. (#29) * feat: add audio device changes detect for windows. (#41) * fix Linux compile (#47) * AudioUnit: Don't rely on category switch for mic indicator to turn off (#52) * Stop recording on mute (turn off mic indicator) (#55) * Cherry pick audio selection from m97 release (#35) * [Mac] Allow audio device selection (#21) * RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (#80) * Allow custom audio processing by exposing AudioProcessingModule (#85) * Expose audio sample buffers for Android (#89) * feat: add external audio processor for android. (#103) * android: make audio output attributes modifiable (#118) * Fix external audio processor sample rate calculation (#108) * Expose remote audio sample buffers on RTCAudioTrack (#84) * Fix memory leak when creating audio CMSampleBuffer #86 ## 3. Simulcast/SVC support for iOS/Android. b0b9fe9 - Simulcast support for iOS SDK (#4) - Support for simulcast in Android SDK (#3) - include simulcast headers for mac also (#10) - Fix simulcast using hardware encoder on Android (#48) - Add scalabilityMode support for AV1/VP9. (#90) ## 4. Android improvements. 9aaaab5 - Start/Stop receiving stream method for VideoTrack (#25) - Properly remove observer upon deconstruction (#26) - feat: Expose setCodecPreferences/getCapabilities for android. (#61) - fix: add WrappedVideoDecoderFactory.java. (#74) ## 5. Darwin improvements a13ea17 - [Mac/iOS] feat: Add RTCYUVHelper for darwin. (#28) - Cross-platform `RTCMTLVideoView` for both iOS / macOS (#40) - rotationOverride should not be assign (#44) - [ObjC] Expose properties / methods required for AV1 codec support (#60) - Workaround: Render PixelBuffer in RTCMTLVideoView (#58) - Improve iOS/macOS H264 encoder (#70) - fix: fix video encoder not resuming correctly upon foregrounding (#75). - add PrivacyInfo.xcprivacy to darwin frameworks. (#112) - Add NSPrivacyCollectedDataTypes key to xcprivacy file (#114) - Thread-safe `RTCInitFieldTrialDictionary` (#116) - Set RTCCameraVideoCapturer initial zoom factor (#121) - Unlock configuration before starting capture session (#122) ## 6. Desktop Capture for macOS. 841d78f - [Mac] feat: Support screen capture for macOS. (#24) (#36) - fix: Get thumbnails asynchronously. (#37) - fix: Use CVPixelBuffer to build DesktopCapture Frame, fix the crash caused by non-CVPixelBuffer frame in RTCVideoEncoderH264 that cannot be cropped. (#63) - Fix the crash when setting the fps of the virtual camera. (#62) ## 7. Frame Cryptor Support. fc08745 - feat: Frame Cryptor (aes gcm/cbc). (#54) - feat: key ratchet/derive. (#66) - fix: skip invalid key when decryption failed. (#81) - Improve e2ee, add setSharedKey to KeyProvider. (#88) - add failure tolerance for framecryptor. (#91) - fix h264 freeze. (#93) - Fix/send frame cryptor events from signaling thread (#95) - more improvements for E2EE. (#96) - remove too verbose logs (#107) - Add key ring size to keyProviderOptions. (#109) ## 8. Other improvements. eed6c8a - Added yuv_helper (#57) - ABGRToI420, ARGBToI420 & ARGBToRGB24 (#65) - more yuv wrappers (#87) - Fix naming for yuv helper (#113) - Fix missing `RTC_OBJC_TYPE` macros (#100) --------- Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> Co-authored-by: David Zhao <dz@livekit.io> Co-authored-by: davidliu <davidliu@deviange.net> Co-authored-by: Angelika Serwa <angelika.serwa@gmail.com> Co-authored-by: Théo Monnom <theo.monnom@outlook.com>

[Mac/iOS] feat: Add RTCYUVHelper for darwin. (webrtc-sdk#28) Cross-platform `RTCMTLVideoView` for both iOS / macOS (webrtc-sdk#40) rotationOverride should not be assign (webrtc-sdk#44) [ObjC] Expose properties / methods required for AV1 codec support (webrtc-sdk#60) Workaround: Render PixelBuffer in RTCMTLVideoView (webrtc-sdk#58) Improve iOS/macOS H264 encoder (webrtc-sdk#70) fix: fix video encoder not resuming correctly upon foregrounding (webrtc-sdk#75). Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com>

hiroshihorie added 6 commits May 18, 2023 17:25

progress

c005df9

enable low-latency video encoding

8ce2df8

fix compile

7892c03

prioritize speed

062d40d

fix required os versions

8ba74bf

maxQP for screensharing

726fb9e

hiroshihorie requested a review from cloudwebrtc May 20, 2023 08:24

hiroshihorie added 3 commits May 21, 2023 21:14

cbr

ea7db41

Update RTCVideoEncoderH264.mm

54aabc3

Update RTCVideoEncoderH264.mm

808223b

hiroshihorie marked this pull request as ready for review May 21, 2023 15:46

format

b4ad069

hiroshihorie requested a review from davidzhao May 21, 2023 15:48

davidzhao approved these changes May 22, 2023

View reviewed changes

cloudwebrtc approved these changes May 22, 2023

View reviewed changes

hiroshihorie added 2 commits May 22, 2023 18:42

clean up

67fb003

resolution alignment

bfd99df

hiroshihorie commented May 22, 2023

View reviewed changes

sdk/objc/native/src/objc_video_track_source.mm Outdated Show resolved Hide resolved

revert resolution alignment

4d21a6f

hiroshihorie merged commit cf25e81 into m104_release May 22, 2023

hiroshihorie deleted the fix-h264-encoder branch May 22, 2023 10:16

cloudwebrtc mentioned this pull request Jun 12, 2023

update to m114 #76

Merged

cloudwebrtc mentioned this pull request May 22, 2024

Update to m125. #119

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve H264 encoder #70

Improve H264 encoder #70

hiroshihorie commented May 18, 2023 •

edited

Loading

davidzhao left a comment

davidzhao May 22, 2023

davidzhao May 22, 2023

hiroshihorie May 22, 2023

davidzhao May 22, 2023

davidzhao May 22, 2023

cloudwebrtc May 22, 2023 •

edited

Loading

cloudwebrtc May 22, 2023

hiroshihorie May 22, 2023

cloudwebrtc May 22, 2023

davidzhao May 22, 2023

cloudwebrtc May 22, 2023

cloudwebrtc May 22, 2023

hiroshihorie May 22, 2023

davidzhao May 22, 2023

cloudwebrtc left a comment

hiroshihorie commented May 22, 2023 •

edited

Loading


		CMTime presentationTimeStamp = CMTimeMake(frame.timeStampNs / rtc::kNumNanosecsPerMillisec, 1000);
		if (CMTimeCompare(presentationTimeStamp, _previousPresentationTimeStamp) == 0) {

Improve H264 encoder #70

Improve H264 encoder #70

Conversation

hiroshihorie commented May 18, 2023 • edited Loading

davidzhao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cloudwebrtc May 22, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cloudwebrtc left a comment

Choose a reason for hiding this comment

hiroshihorie commented May 22, 2023 • edited Loading

hiroshihorie commented May 18, 2023 •

edited

Loading

cloudwebrtc May 22, 2023 •

edited

Loading

hiroshihorie commented May 22, 2023 •

edited

Loading