Passthrough yuv format for ffmpeg-h264 decoder when possible #8002
Conversation
Wumpf
added
🚀 performance
|
We have the fix 🚀 https://github.com/nathanbabcock/ffmpeg-sidecar/releases/tag/v2.0.2 |
| if sps_units.next().is_some() { | ||
| // This is rather strange. Must mean that some pictures refer to one SPS and some to another! | ||
| // We don't know what to do with this. | ||
| re_log::trace_once!("Found more than one sequence parameter set (SPS) in the AVCC box of {debug_name}."); |
There was a problem hiding this comment.
I suggest we use warn in debug builds. We could add some helper for that in re_log
There was a problem hiding this comment.
after refactoring the whole thing to a method and - sparked by that - modularizing stuff more, I decided to just always warn. I'd rather have people complain about this than not finding these issues in our understanding!
| /// Semantics are defined in [ITU-T H.264 (04/2017)](http://wikil.lwwhome.cn:28080/wp-content/uploads/2018/08/T-REC-H.264-201704%E8%8B%B1%E6%96%87.pdf) | ||
| #[derive(Debug)] | ||
| #[allow(dead_code)] | ||
| pub struct H264Sps { |
There was a problem hiding this comment.
I assume this appears in all H.264 stream, regardless of container format? i.e. that is why it belongs in re_video, and not re_mp4?
There was a problem hiding this comment.
Correct, the SPS is part of the h264 spec (as linked in the comment). So this will still be useful when streaming in data as we'll want to pass through the AVCC box (which typically contains the SPS among other things). At least I think at this point that we want to insist on that rather than consuming AnnexB (I recently read up some more on how this works in WebCodecs which btw. is a great blueprint for our mux-free/streaming api ;-) [.........])
Wumpf
force-pushed
the
andreas/h264-yuv-passthrough
branch
from
dc59d9e to
383e6c6
Compare
What
ffmpegCLI #7607Recognizing ahead of time what format a piece of encoded data is unfortunately not that easy. To accomplish that I fell into the rabbit hole of parsing the Sequence Parameter Set (SPS). Which will hopefully be useful for other things in the future!
Other than that, it turned out that determining the color space & color range are again different cans of worms, so I'm being now a lot more explicit with ffmpeg on what we expect.
Which.. is yet another rabbit hole! Pretty much no matter what I tried I couldn't replicate exact colors some sample videos compared to VLC (unlike we did with AV1 where we have a lot more solid grasp on all these things and get satisfying parity!). But also here different video players seem to do slightly different things, so I decided to not follow this thread further for the moment as this whole affair already cost me a lot more time that I wanted to invest originally.
Partially disabled unfortunately since we need this fix:(note that this uses 422, the video source material is 420 though)
Fixes slow 4k video handling:
Before:
After:
Checklist
mainbuild: rerun.io/viewernightlybuild: rerun.io/viewerCHANGELOG.mdand the migration guideTo run all checks from
main, comment on the PR with@rerun-bot full-check.