Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Account for video with DTS shift and resulting negative dts values #16

Merged
merged 2 commits into from
Nov 13, 2024

Conversation

Wumpf
Copy link
Member

@Wumpf Wumpf commented Nov 12, 2024

This fixes

Other mp4 parsers/data processors apply similar things:

... still this begs the question:

  • is this speced? couldn't find anything
  • this blurs the line a bit of what "the raw PTS/DTS" is. Rerun already offsets by min PTS. Maybe we should do that as well inside re_mp4. This way, ffprobe's output would always match ours (it does not right now as it's shifted by the min PTS, see example below)

First 6 frames in Rerun of the video from rerun-io/rerun#8098:

image

And for comparison the same frames from ffprobe, using ffprobe -of json -show_packets -select_streams v ../internal-test-assets/dtsbroken.mp4 > dtsbroken.json:

 {
            "codec_type": "video",
            "stream_index": 0,
            "pts": 0,
            "pts_time": "0.000000",
            "dts": -2002,
            "dts_time": "-0.066733",
            "size": "51711",
            "pos": "24912",
            "flags": "K__"
        },
        {
            "codec_type": "video",
            "stream_index": 0,
            "pts": 4004,
            "pts_time": "0.133467",
            "dts": -1001,
            "dts_time": "-0.033367",
            "size": "7564",
            "pos": "79488",
            "flags": "___"
        },
        {
            "codec_type": "video",
            "stream_index": 0,
            "pts": 2002,
            "pts_time": "0.066733",
            "dts": 0,
            "dts_time": "0.000000",
            "size": "2192",
            "pos": "87052",
            "flags": "___"
        },
        {
            "codec_type": "video",
            "stream_index": 0,
            "pts": 1001,
            "pts_time": "0.033367",
            "dts": 1001,
            "dts_time": "0.033367",
            "size": "1911",
            "pos": "89244",
            "flags": "___"
        },
        {
            "codec_type": "video",
            "stream_index": 0,
            "pts": 3003,
            "pts_time": "0.100100",
            "dts": 2002,
            "dts_time": "0.066733",
            "size": "1168",
            "pos": "91155",
            "flags": "___"
        },
        {
            "codec_type": "video",
            "stream_index": 0,
            "pts": 8008,
            "pts_time": "0.266933",
            "dts": 3003,
            "dts_time": "0.100100",
            "size": "7874",
            "pos": "94674",
            "flags": "___"
        },

Note that this matches now for this video!


However, on "timestamped bunny" we still have the min pts discrepancy which we account for upon query in Rerun (unchanged from before/after this PR!):
image

        {
            "codec_type": "video",
            "stream_index": 0,
            "pts": 0,
            "pts_time": "0.000000",
            "dts": -512,
            "dts_time": "-0.033333",
            "duration": 256,
            "duration_time": "0.016667",
            "size": "190349",
            "pos": "48",
            "flags": "K__"
        },
        {
            "codec_type": "video",
            "stream_index": 0,
            "pts": 1024,
            "pts_time": "0.066667",
            "dts": -256,
            "dts_time": "-0.016667",
            "duration": 256,
            "duration_time": "0.016667",
            "size": "13884",
            "pos": "190397",
            "flags": "___"
        },
        {
            "codec_type": "video",
            "stream_index": 0,
            "pts": 512,
            "pts_time": "0.033333",
            "dts": 0,
            "dts_time": "0.000000",
            "duration": 256,
            "duration_time": "0.016667",
            "size": "3790",
            "pos": "204281",
            "flags": "___"
        },
        {
            "codec_type": "video",
            "stream_index": 0,
            "pts": 256,
            "pts_time": "0.016667",
            "dts": 256,
            "dts_time": "0.016667",
            "duration": 256,
            "duration_time": "0.016667",
            "size": "2169",
            "pos": "208071",
            "flags": "___"
        },

It works, so I'd say we leave it for now 🤔 . But something that might came back to haunt us!

@Wumpf Wumpf force-pushed the andreas/negative-dts branch from 5e77c75 to 57d0bde Compare November 12, 2024 18:03
@emilk
Copy link
Member

emilk commented Nov 12, 2024

this blurs the line a bit of what "the raw PTS/DTS" is. Rerun already offsets by min PTS. Maybe we should do that as well inside re_mp4. This way, ffprobe's output would always match ours (it does not right now as it's shifted by the min PTS, see example below)

Yeah, it would be nice if re_mp4 did this so that complexity was moved out of Rerun 🤔

@Wumpf Wumpf added enhancement New feature or request include in changelog Will appear in CHANGELOG.md labels Nov 13, 2024
@Wumpf Wumpf merged commit 8fa1613 into main Nov 13, 2024
5 checks passed
@Wumpf Wumpf deleted the andreas/negative-dts branch November 13, 2024 09:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request include in changelog Will appear in CHANGELOG.md
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants