RaspiVid: motion vector synchronization is awry

[malc0]

It seems the motion vectors always arrive a frame late relative to the video, which is odd considering they ought to be available before the fully encoded video frame is.

The below image shows the frames from a short recording made using raspivid -ss 5000 -fps 20 -t 700 -o moo.h264 -x moo.imv, while the bottom image shows plots of the corresponding SAD motion vector coefficients (files available at http://sb476.user.srcf.net/late_mvs/moo.h264 and http://sb476.user.srcf.net/late_mvs/moo.imv ).

In the recording I waved a pen in front of the camera, seen in frame 4 of the recording, but not apparent until frame 5 (and residually 6) of the motion vectors.

I had added printf("timedelta %lld pts %lld flags %x\n", vcos_getmicrosecs64()/1000-base_time, buffer->pts, buffer->flags); near the top of encoder_buffer_callback(), which after some formatting and annotation of frames (F) versus motion vectors (MV) gives:

(recording starts)
timedelta 0     pts 0                           flags 20 
timedelta 2     pts 0                           flags 24 
timedelta 270   pts 743430883                   flags c         F 1 
timedelta 273   pts 0                           flags 84                MV 1 
timedelta 306   pts 743480704                   flags 4         F 2 
timedelta 308   pts 0                           flags 84                MV 2 
timedelta 356   pts 743530524                   flags 4         F 3 
timedelta 357   pts 0                           flags 84                MV 3 
timedelta 414   pts 743580344                   flags 4         F 4 
timedelta 417   pts 0                           flags 84                MV 4 
timedelta 461   pts 743630164                   flags 4         F 5 
timedelta 463   pts 0                           flags 84                MV 5 
timedelta 516   pts 743679984                   flags 0         F 6 
timedelta 519   pts -9223372036854775808        flags 4         F 6 
timedelta 522   pts 0                           flags 84                MV 6 
timedelta 567   pts 743729805                   flags 4         F 7 
timedelta 572   pts 0                           flags 84                MV 7 
timedelta 620   pts 743779624                   flags 4         F 8 
timedelta 623   pts 0                           flags 84                MV 8 
timedelta 673   pts 743829445                   flags 0         F 9 
timedelta 675   pts -9223372036854775808        flags 4         F 9 
timedelta 677   pts 0                           flags 84                MV 9 
timedelta 725   pts 743879264                   flags 0         F 10 
timedelta 727   pts -9223372036854775808        flags 4         F 10 
timedelta 729   pts 0                           flags 84                MV 10 
timedelta 778   pts 743929084                   flags 0         F 11 
timedelta 781   pts -9223372036854775808        flags 4         F 11 
timedelta 783   pts 0                           flags 84                MV 11 
timedelta 823   pts 743978905                   flags 0         F 12 
timedelta 826   pts -9223372036854775808        flags 4         F 12 
timedelta 828   pts 0                           flags 84                MV 12 
timedelta 877   pts 744028725                   flags 0         F 13 
timedelta 880   pts -9223372036854775808        flags 4         F 13 
timedelta 882   pts 0                           flags 84                MV 13 
timedelta 929   pts 744078544                   flags 0         F 14 
timedelta 932   pts -9223372036854775808        flags 4         F 14 
timedelta 935   pts 0                           flags 84                MV 14 
timedelta 979   pts 744128365                   flags 0         F 15 
timedelta 982   pts -9223372036854775808        flags 4         F 15 
timedelta 985   pts 0                           flags 84                MV 15
(recording ends)

Two things are apparent: MV buffers have no timestamp, but they always arrive immediately following the video buffer. It can be seen that MV 5 arrives 49 ms after F 4 (with MV 4 and F 5 intervening), when inspection of the images above suggest they ought to arrive together, or even with the motion vectors preceding the encoded video.

While attaching correct timestamps to MV buffers would help make this clear, it seems there is some strange delay going on in the buffers' delivery.

[ghollingworth]

Must admit I'm not 100% sure why you are getting the results you have... I did specifically push out a frame of 0 motion vectors for any I-Frames (i.e. Frame 1) but Frame 2 should give you the vector between frame 1 and frame 2

[6by9]

MVs are after the encoded data as video_encode appears to have been set to enforce that ordering (output_type only gets set to EV_OUTPUT_SIDEINFO after the first non-header bytes packet). That's probably the behaviour needed for eede. It could be reversed as EEDE is not officially supported.

I think the example is a bad one. The pen coming into frame will NOT produce sensible motion vectors as there in no sensible block that is black to use for predicting those macroblocks.
As the pen EXITS the frame (ie frame 4-5 transition, giving MV5), then it can use some of the surrounding grey pixels to reinstate the "no-pen" state, and the following frame there is a small amount of residual as it improves that prediction.

So I think this is actually the correct behaviour, other than it being nice if the MVs were before the encoded data.
Timestamps for the MVs may be possible, but it would need bit of thought and may require the encode data to come first (id->data_hdr will then be correct for the buffer just produced, so you can get the timestamp)

[malc0]

Perhaps this is down to me misunderstanding how the sum of differences image is arrived at (I do accept that the displacement vectors will be nonsense), but surely the difference image between video frames 3 and 4 (no pen -> pen) should not be completely smooth, as in the contrast stretched plot of MV frame 4 below?

A related observation is that if I capture scene A, stop the recording, then point the camera at scene B (different) and do a second capture, MV buffer 2 from the second capture contains differences related to the scene A capture, with no scene B details appearing until MV buffer 3.

[6by9]

@gsh Back to you as you may remember more of how the CME operates. Your code does all look sane on first glance - if I can then I'll try running it up with a debugger to see if there is something wrong in the sequencing.
Whether the results are exactly as desired here depends on how the CME is designed to operate. If it is trying to find the best match for encoding, then producing a near zero result when there is no good match seems valid.

[ghollingworth]

Yes, the CME works by doing SADs (sum of absolute differences) at every position within its search area then it returns the lowest vector as long as the SAD is less than some threshold, otherwise it just returns zero (a zero vector encodes to a small bit stream so it's just trying to punish larger vectors)

So the difference between two images may be a full set of vectors of zeros because it couldn't find a suitable match in the previous image for the macroblock (which if the previous image is all the same then you'd expect a zero vector to be as good as any other vector)

Gordon

[malc0]

Right, but clearly in the most recent image the blocks occupied by the pen (which is essentially NW to SE diagonal across the frame) do not have zero values, they have intensities very similar to their neighbours in non-pen areas, as if it were encoding the difference of two frames neither of which have the pen.

[ethanol100]

I want to confirm the issue. I have modified raspistill to use the still_port and could change the position of an object in one image.

Here after the 4th image the position was changed, but only in the 6th frame the data shows the corresponding motion. Only motion vector differnet from zero are displayed.

Additionally the the second motion buffer seems wrong to me. Could an old, not yet released buffer from the first frame creates this additional buffer?

For reference the recorded data:
http://www2.geo.uni-bonn.de/members/kroener/test_imv.h264
http://www2.geo.uni-bonn.de/members/kroener/test_imv.imv

[6by9]

I've found something that looks a little fishy in the code and I think would cause 1 frame of latency. Email discussion is ongoing between Dom, Gordon, and myself as the solution isn't immediately obvious without crippling performance. I can't remember (may have never known) sufficient details of the H264 hardware block to say for certain how it is meant to work around this bit.

[6by9]

@ethanol100: could you post your test app somewhere? I have a potential fix, but currently no easy way of testing.

[ethanol100]

I modified raspivid not raspistill...
https://github.com/ethanol100/userland/tree/stopMotion
The preview does not show the right image yet.
You can use "./raspivid --use-stills -t 0 -k -o still.h264 -v" then each time you press enter, a new frame is captured from the still port, I had to change the still port format encoding to MMAL_ENCODING_I420.
(added ./build/bin/raspivid for quick testing.)

[6by9]

Thank you. I'll try it out as soon as I get a chance (possibly tomorrow morning).

[6by9]

Looks like we have a solution. The original code was reading the CME output memory without having waited for it to complete, hence got the results for the previous frame. The synchronisation required for H264 encode is done with a hardware interlock on the results.
I've punted a patch over to be sanity checked and released, and that will hopefully solve this issue.

Thanks again @ethanol100 for the test case - saved me a lot of time. Although I do see a changing in FOV for each of the captures - is that expected? I'm guessing switching between sensor modes (probably the 1080P cropped and 5MPix modes), but didn't stop to investigate.

[popcornmix]

Firmware has been updated. Please run rpi-update and test again.

[ethanol100]

This fixed the issue for me. Thanks for the quick fix and for the fast firmware update.

[popcornmix]

@malc0 okay for you?

[malc0]

Yes, both of my tests pass well. Thanks for taking the time to look in to (and fix) this.

[popcornmix]

Thanks 6by9!

[konstantins-tracxpoint]

I think motion vectors is completely "unsynchronized"
reasons:

1. no pts in motion vectors packet(see in raspivid)
2. My test via OMX encoder (rpi-encode-yuv) shows what
   motion vectors delay is: delta = (encoder output packet time - encoder input packet time)
   (but can change with some motion)

[6by9]

@konstantins-tracxpoint Please open a new issue (referencing this one) with as much detail as possible.
This one has been closed for 4 years, and will have a load of unnecessary baggage.

You also reference a test app that no one else knows anything about - more information please.