Seeking frame sync solutions

[mprib]

I am currently in the process of updating pyxy3d to calibrate and triangulate with pre-recorded footage. My internal target for having that available is end-of-November.

One obstacle to the usefulness of this is that it requires input footage start and stop on the same frame, something that I have found is a challenging problem to automate when using footage captured from cameras such as GoPros/iPhones.

I'm hoping to better understand what tools currently exist that enable this kind of footage synchronization. I would like to have something to point people toward/use myself/contribute to.

---

@rgov : I'm tagging you here as you had just expressed an interest in this and I'm wondering what your plans for footage sync may be.

[rgov]

Other software I've seen just lets you step frame by frame to line up something like a clapper board and then record the offset for each camera. You can show a precise reference clock on an iPhone, say -- those screens get up to 120Hz.

If you have audio tracks, you can match up the waveforms and compute the offset, like this project does. I think I saw another project for this but don't remember the name.

I experimented earlier this year with using a strobing light and then searching for a pattern in the luma values, but that's for future exploration.

[mprib]

Very glad to know about the linked project, thank you. I'm coming to this from a biomechanics background where just capturing someone walking on a treadmill would use 10 cameras, so creating a fully automated sync is definitely on my radar as a long-term need.

[mprib]

Documenting here another possible solution to the frame sync:
https://github.com/allisonnicoledeal/VideoSync

[rgov]

I ran an experiment filming with two non-synchronized cameras, of approximately the same model. Stepping through the first few minutes, I found them to have an approximately 30 frame offset, so I padded one video with 30 frames of black to bring them into alignment.

However, after an hour of footage the frames were again noticeably out of alignment by about 53 frames.

After two hours they are out of alignment by 24 frames.

Therefore alignment of the videos seems less straightforward than finding an offset.

[mprib]

Drift is definitely something that I have been concerned about. My thought has been to enforce synchronization for both the start and stop frames, so there might need to be something like a sound cue at the beginning and end. With those known, estimated times for each frame capture can be assigned. Without the same frame counts, there will be a time misalignment and then during processing the synchronizer will drop time-mismatched frames (EDIT: it won't actually drop frames, it will just insert a blank frame within a batch of synchronized frames).

This won't be perfect, but I suspect that this will average out to keep things closely in sync in the presence of drift. The synchronizer is already coded up because dropped frames were a real problem when pulling from multiple webcams at once, so that part of the workflow is already baked in.

[rgov]

Consider allowing the user to add as many sync points-in-time as they want. Display it as a table with editable cells, where you can enter frame numbers:

	Video 1 Frame No.	Video 2 Frame No.	...
t0	6554	6524
t1	7397	7365

Then you could linearly interpolate and insert your blank (or duplicated) frames as necessary.

I know that just start and end won't be enough for my videos given the offsets I just measured.

[lambdaloop]

I started to explore synchronization with gopros and smartphone using sound as well, and ran into the same issue as @rgov . The drift between gopro cameras seems to be inconsistent, and I've tracked one possible reason down to the length of time between chunks of videos. It took about 0.05s to save a video, but would not be consistent across different gopros. This leads to sudden dips or increases in the time offset of different videos.

I've started exploring syncing the chunks to each other directly, which may improve the syncing within gopros or from gopros to a long smartphone recording, which shouldn't have these sudden jumps.

I calculated the drift between smartphone and gopros across time and it seems to be linearly decrease, in which case maybe some linear correction for frame numbers could work.

Here are some figures showing the shift in temporal offset calculated every 30 seconds in an hour long video.


I guess if you're thinking of syncing videos at a frame level and experience something like this, you may need to do a finer-grain sync than start/end

[mprib]

Thank you for this information. It's incredibly useful as I think through future stages of the project. I want to make sure I'm understanding the information here fully...

• How is gold-standard time here being assessed? I'm guessing perhaps the cameras are filming a common digital clock...

• Just rephrasing the "chunk" part to confirm my understanding: on a single gopro after 500 frames (or whatever) the gopro pushes frames into long term storage, but then when it rejoins to grab the 501st frame there is now a noisy time delay because of the processing demands of saving?

I really appreciate you offering your insights here! Your work on Anipose paved the way for this project to get underway. If you have any thoughts or suggestions about any aspects of pyxy3d I'd be very interested in hearing them! The final stages of resolving GUI stuff is dragging on, but my hope is that it will be ready to tinker around with in a stable way within days.

[lambdaloop]

Ah sorry, I should have provided more details here. The videos are first synced within 0.2 seconds of each other, then I chunk each video in 30 second chunks and estimate the offset by cross-correlating the audio. This is fine in chunks, but I guess at the hour level it fails due to drift.

Yes, that's what I mean about the GoPro chunks. It splits long videos into separate files and there is a pause between saving them. To be honest, I'm not entirely sure if this is the cause, or if the GoPro clock is just messed up somehow?

Here is the code I used to make this:
https://gist.github.com/lambdaloop/e2c50f9bfbadcae8407d9c1a89b9052b

[lambdaloop]

Unrelated to this, but I'm quite excited about pyxy3d as a tool for visual and interactive calibration! I look forward to your version with offline video acquisition support, I'm excited to try it and see how it is! I think some way to calibrate the cameras when you're right there at the rig is crucial for recording 3D tracking (especially in experimental setups) at scale.

[mprib]

Thank you for your enthusiasm! It means a lot...

In advance of putting together more formal documentation I've just uploaded a couple of videos that walk through:

• the project structure for file naming and placement: https://youtu.be/dvosFrFVDsU
• a general walk through of the workflow in the GUI: https://www.youtube.com/watch?v=XDfuQ_7m1oQ

PyPI now has the latest version. When installed in the local environment it can be launched with pxyy3d on the command line.

The video walk through notes some little quirks here and there that I plan to keep hacking away at over the coming days.

---

Regarding the drifting time-offsets, I am thinking about the suggestions from @rgov about multiple checkpoints for synchronization (which is seems could be done with your consecutive cross correlation). The current workflow can incorporate a file called frame_time_history.csv that contains a table of ports and frame_times for each frame that was read. The synchronization then is all based off those time stamps. I'm just wondering if there is a way to use the method of recalculating offsets across chunks to impute the frame time history to keep things mostly in sync...

Though a hardware solution to the synchronization definitely seems more ideal...

[mprib]

@lambdaloop,

Would you be willing to issue a Pull Request with your time-offset code into the /dev folder at the root of the project? I'd like to keep the contribution history clear and hope to start experimenting with automating the creation of the frame_time_history.csv file. It seems like your code has already done the heavy lifting on that and I just need to figure out how to knit things together...

Related to this, do you have guidance for what kind of audio works well for the approaches taken in the code?

[lambdaloop]

In animal labs, people mostly use Basler / FLIR cameras and set up an external trigger from the same cable, so that the cameras are guaranteed to be synced and recording at a constant framerate. You could use Bonsai or something like https://github.com/ksseverson57/campy to record from those cameras.

If you have funding and your setup allows for wiring a common external trigger to each camera, I think this would be the safest option.

[mprib]

Thank you for pointing toward tools being used out in the world. A good hardware solution seems like a much more robust approach than trying to clean things up in post processing with software. The challenges of this remind me that I should probably keep the scope of this package a little more tight (i.e. BYOSF: Bring Your Own Synchronized Frames)