Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Keyframe Selection: Add multi-threading and improve overall performances #1512

Merged
merged 5 commits into from
Aug 25, 2023

Conversation

cbentejac
Copy link
Contributor

@cbentejac cbentejac commented Aug 18, 2023

Description

This PR speeds up the selection of keyframes on two different accounts:

  • During the computation of the sharpness scores, the sliding window used to be moved pixel by pixel, which resulted in very accurate sharpness scores but also expensive computations. As the sharpness score can afford losing a bit of accuracy, the sliding window is now moved across the image by steps of windowSize / 4, with windowSize the size of the sliding window.
  • For the computation of all the scores, blocks of frames whose size depends on the number of available threads are computed and then processed in dedicated threads. Blocks are overlapping, since the optical flow requires the current frame N as well as the previous frame N-1.

It relates to alicevision/Meshroom#2161 and should only be merged once #1510 has been integrated into develop.

Features list

  • Use a larger step for the sliding window of the sharpness score computation, effectively reducing the computation time for the sharpness score;
  • Add multi-threading for the computation of all scores.

Implementation remarks

The parallelization system using chunks that is used in Meshroom cannot be used to speed the keyframe selection up, because only the step of the score computation can be parallelized: once all the scores have been computed, they need to be put back all together sequentially for the keyframes to actually be selected and this process cannot be split.

@cbentejac cbentejac added this to the 3.2.0 milestone Aug 18, 2023
@cbentejac cbentejac self-assigned this Aug 18, 2023
@cbentejac cbentejac force-pushed the dev/improveKeyframeSelectionPerf branch 3 times, most recently from a557071 to 7fb4c60 Compare August 24, 2023 20:51
@cbentejac cbentejac marked this pull request as ready for review August 25, 2023 07:16
The pixel aspect ratio wasn't written in the output images' metadata if
they were written as JPEG because of an issue in OIIO. With the release
of version 2.4.13.0 of OIIO, this problem is solved and the correct pixel
aspect ratio value can now be propagated for JPEG images.
The sliding window for the sharpness computation was moving pixel by pixel,
which was costly computation-wise for limited extra accuracy.

The window is now moved windowSize / 4 by windowSize / 4 pixels, which
reduces the number of computations for an equivalent accuracy.
Add a new argument to set the minimum number of frames that must be
processed by a thread for it to be spawned, and the maximum number of
threads that can be used.
@cbentejac cbentejac force-pushed the dev/improveKeyframeSelectionPerf branch from 7fb4c60 to b43dbbf Compare August 25, 2023 13:55
@mugulmd mugulmd merged commit 4a42790 into develop Aug 25, 2023
@mugulmd mugulmd deleted the dev/improveKeyframeSelectionPerf branch August 25, 2023 14:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants