Add a way to reduce latency

[GameParrot]

Sometimes the latency randomly gets high and doesnt go back down until the page is reloaded. Could a way to reload the video or if it gets behind jump ahead be added?

[was2]

I see this latency behavior as well

[m1k1o]

Have you as well tested with Firefox? Or is the latency different between chrome and Firefox (as host browser).

[GameParrot]

Have you as well tested with Firefox? Or is the latency different between chrome and Firefox (as host browser).

still some noticeable latency on firefox

[GameParrot]

Have you as well tested with Firefox? Or is the latency different between chrome and Firefox (as host browser).

Can confirm that Chrome has much higher latency than other browsers

[m1k1o]

Then I guess its related to the internal audio jitter buffer that chrome uses. It cannot be modified. We don't have any solution for that currently.

[mbattista]

https://github.com/VLprojects/webrtc-issue-detector

this project could be integrated to have an easier look what could be problematic.
Also the source could be used to detect some issues and reload the page or show a message about the latency

[ehfd]

Using two webrtcbin objects in GStreamer (and for Neko, separate Pion streams for audio and video) works to reduce a pretty good chunk of latency (capable of reducing 150-200 ms of latency to 10-20 ms). It seems that there is an inherent latency in Chromium and Firefox which is introduced when audio and video are sent together with WebRTC.

This is thus, a browser-side issue where Chromium and Firefox introduce more latency to sync video and audio into the

Explained very elaborately in:
https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1261

However, the web interface needs to check the status of both the audio WebRTC and video WebRTC streams and respond when either one fails (then reload or fail the whole instance if the failure doesn't improve in either one of the streams).

It's very important to emphasize that the web interface must ensure both video and audio connections are established and retained. Else, there are frequently scenarios where the video is seen but no audio is to be heard.

selkies-project/selkies-gstreamer#7

selkies-project/selkies-gstreamer#109

[ehfd]

Moreover, the possible usage of https://quic.video through WebTransport is a good possibility to eliminate the exposure of multiple host ports/usage of TURN servers for WebRTC and reduce latency even further.

But the focus is to use separate streams for audio and video, then check that both streams are alive in the web interface, then reload or fail the whole instance. This will reduce ~95% of the current latency at the cost of using two times as many ports (look below) for WebRTC. But it's worth it.

[ehfd]

Relevant for the web interface redesign for #358.

[ehfd]

https://groups.google.com/g/discuss-webrtc/c/ZvAHvkHsb0E
https://groups.google.com/g/discuss-webrtc/c/yExd2Kou54c
https://webrtc.googlesource.com/src/+/refs/heads/main/docs/native-code/rtp-hdrext/abs-capture-time/

[ehfd]

A few more updates (of which are more important):

https://webrtc.googlesource.com/src/+/main/docs/native-code/rtp-hdrext/playout-delay/
The above extension is extremely important to eliminating video latency in WebRTC.

https://multi.app/blog/making-illegible-slow-webrtc-screenshare-legible-and-fast
https://www.rtcbits.com/2023/05/webrtc-header-extensions.html

Moreover, it seems that having two separate WebRTC sessions isn't a strict requirement to disable A/V sync. It actually looks specific to the GStreamer webrtcbin implementation.
Putting audio, video, and DataChannel into one session, while disabling all sync to minimize latency, will save a TURN port per session.

Observe how this would play out with Pion...

Like:

v=0
o=- 2 IN IP4 1.1.1.1
t=0 0
a=group:BUNDLE 0 1 2 3
a=fingerprint:sha-256
a=setup:actpass
m=audio x UDP/TLS/RTP/SAVPF 111 63
c=IN IP4 0.0.0.0
a=rtcp:x IN IP4 0.0.0.0
a=mid:0
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=sendonly
a=msid:id audio
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:63 red/48000/2
a=rtcp-fb:63 transport-cc
a=fmtp:63 111/111
a=ptime:10
m=video x UDP/TLS/RTP/SAVPF 96 97 101 102 98
c=IN IP4 0.0.0.0
a=rtcp:x IN IP4 0.0.0.0
a=mid:1
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:12 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=sendonly
a=msid:id video
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:101 H264/90000
a=rtcp-fb:101 transport-cc
a=rtcp-fb:101 ccm fir
a=rtcp-fb:101 nack
a=rtcp-fb:101 nack pli
a=fmtp:101 level-asymmetry-allowed=1;packetization-mode=1;sps-pps-idr-in-keyframe=1;profile-level-id=42e01f
a=rtpmap:102 rtx/90000
a=fmtp:102 apt=101;rtx-time=125
m=application x UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=mid:2
a=sctp-port:5000
a=max-message-size:262144
m=audio x UDP/TLS/RTP/SAVPF 111
c=IN IP4 0.0.0.0
a=rtcp:x IN IP4 0.0.0.0

Also:
urn:ietf:params:rtp-hdrext:sdes:mid: https://datatracker.ietf.org/doc/html/draft-ietf-mmusic-sdp-bundle-negotiation-12