Implement (optional) buffering #2464
Conversation
rom1v
force-pushed
the
buffering.10
branch
2 times, most recently
from
8f80c7d to
b660a24
Compare
|
I improved the smoothness on start, reworked a bit and fixed some bugs. |
rom1v
force-pushed
the
buffering.10
branch
3 times, most recently
from
1d2c85b to
b454af6
Compare
This avoids to use the SDL timer API directly, and will allow to handle generic ticks (possibly negative).
The function sc_cond_timedwait() accepted a parameter representing the max duration to wait, because it internally uses SDL_CondWaitTimeout(). Instead, accept a deadline, to be consistent with pthread_cond_timedwait().
To fix a data race, commit 5caeab5 called video_buffer_push() and video_buffer_consume() under the v4l2_sink lock. Instead, use the previous_skipped indication (initialized with video buffer locked) to lock only for protecting the has_frame flag. This enables the possibility for the video_buffer to notify new frames via callbacks without lock inversion issues.
Add a scrcpy-specific prefix.
The current video buffer only stores one pending frame. In order to add a new buffering feature, move this part to a separate "frame buffer". Keep the video_buffer, which currently delegates all its calls to the frame_buffer.
Currently, a frame is available to the consumer as soon as it is pushed by the producer (which can detect if the previous frame is skipped). Notify the new frames (and frame skipped) via callbacks instead. This paves the way to add (optional) buffering, which will introduce a delay between the time when the frame is produced and the time it is available to be consumed.
To minimize latency (at the cost of jitter), scrcpy always displays a frame as soon as it available, without waiting. However, when recording (--record), it still writes the captured timestamps to the output file, so that the recorded file can be played correctly without jitter. Some real-time use cases might benefit from adding a small latency to compensate for jitter too. For example, few tens of seconds of latency for live-streaming are not important, but jitter is noticeable. Therefore, implement a buffering mechanism (disabled by default) to add a configurable latency delay. PR #2417 <#2417>
Add --display-buffer and --v4l2-buffer options to configure buffering time.
The clock rolling sum is not trivial. Test it.
Output buffering and clock logs by disabling a compilation flag.
The first frames are typically received and decoded with more delay than the others, causing a wrong slope estimation on start. To compensate, assume an initial slope of 1, then progressively use the estimated slope.
|
Public note to self. I am curious to look more into v4l2loopback performance limitations. I noticed v4l2loopback appears less stable - as well possibly lower - framerate compared to the scrcpy window, with no buffering in both cases. I became curious is there some performance issue with scrcpy's v4l2loopback implementation, or with v4l2loopback itself, that can account for this discrepancy I am noticing with no buffering in both cases? Also, irrelevant, curious about the v4l2loopback-ctl set-fps command and how it can interact with scrcpy --max-fps option. |
Which player do you use to play the v4l2 stream? |
|
obs, discord, ffplay. they all give similar results. since the above note started so unscientifically, I feel like apologizing... there are a lot of configuration options I am experimenting with... I may be blaming v4l2loopback for something unrelated... I was fiddling more this morning and started removing options one-by-one from my main script I use --v4l2-sink with... first I removed the --no-display option and compared the video players to the scrcpy window, and the same stability issue was present in both... that's the first thing I should have tried, sorry. then I removed the --turn-screen-off option and the stability of both the window and v4l2loopback stream seemed to return to normal. this is still just my perception, but whatever I'm noticing may have to do with the --turn-screen-off option, not anything to do with v4l2loopback. apologies. |
hello, does this mean that when recording a video, scrcpy will get the timestamp corresponding to the frame on the device (mobile phone), and then encode the video according to this timestamp? How can I get this exact timestamp? |
Yes.
Grep |
To minimize latency, scrcpy always displays a frame as soon as it is available, without waiting. This design decision is (on purpose) at the cost of jitter: the delay between frames is not preserved.
In an ideal world, every frame produced by the device would be displayed some constant time later on the device (for simplicity, the schema assumes a constant framerate):
However, in reality, the delay is not constant; the encoding time, transfer time and decoding time may vary from frame to frame:
In practice, this is not a problem for many real-time use cases. It is slightly noticeable when playing a video on the device (or anything involving regular movement) while mirroring.
On recording (
--record), scrcpy captures the timestamps on the device and write them in the resulting file, so that the playback is correct without jitter.However, some real-time use cases may benefit from adding a small latency to compensate for jitter too. For example, few tens of milliseconds of latency for live-streaming are not important, but jitter is noticeable.
To support this, the principle is to delay the frames to compensate for jitter (notice how the blue circles have the same spacing as the green circles):
This PR implements optional buffering:
Display buffering and v4l2 are independent, so that it is possible for example to control it with minimal delay but stream it with a larger delay.
In practice, here is a graph showing the difference in smoothness (after a small initialization phase) with a buffer of 50ms on a wireless connection:
(graph_old_impl)
Fixes #2417