Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Video decoding tests are never run on CI #8162

Open
NicolasHug opened this issue Dec 11, 2023 · 1 comment
Open

Video decoding tests are never run on CI #8162

NicolasHug opened this issue Dec 11, 2023 · 1 comment

Comments

@NicolasHug
Copy link
Member

NicolasHug commented Dec 11, 2023

The testing of the video decoders is in a catastrophic state. Almost all tests for read_video, VideoReader or the GPU decoder are being skipped on our CI, for one reason or another.

  • The GPU tests are never run because we never build with support for the nvidia lib. (I guess it's sort of OK considering the state of that GPU decoder itself... we should probably just remove that decoder entirely.)

  • The other tests (read_video and VideoReader) are never run either. @pmeier's [DEBUG] video extension not available in CI #8157 suggests that the video ops cannot be loaded with the following error:

Failed to load video Python extension: '/lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /opt/conda/envs/ci/lib/./libopenh264.so.5)'

Although for some Python versions, we don't even get an error message.

  • The tests in test_io.py are being run. They only test read_video on the pyav backend AFAICT.

The lack of test for those decoders isn't new and has been going on for a LONG time. In #7646 we already noticed part of the problem.

Part of the reason this hasn't been flagged earlier is because our tests are setup incorrectly: they don't tell you if they can't be run! If for some reason the ops cannot be loaded, or if pyav isn't present, or if there was a problem with the ffmpeg build, then the tests are simply skipped. I understand this is because we can't be testing those APIs on all platforms, but one bad consequence is that when the build fails (and it FAILS), then the tests are just skipped and CI is still happy with that, so we never know. We think the tests are green when in reality they were never run.

So, beyond bringing those tests back, we have to find a way to be properly alerted when those tests aren't run when we expect them to be run.

On #8161 we have decided to at least stop pretendying, and explicitly excluded those files from being tested until the situation gets resolved. We should probably start with bringing back the read_video tests.

@NicolasHug
Copy link
Member Author

See #8211 for related investigations. This relates to ffmpeg6 but it's probably relevant for all ffmpeg versions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant