How to run benchmark

[howjmay]

Hi I want to run some improvements. However, I didn't find the way to run benchmark. May I ask is there a standard way to run benchmark in this project?

[adamjw24]

Hi.

So that's a complicated question and there is not really a standarized procedure, but I'd like to give you some general guidance.

What kind of improvements are you looking into? In the end we use the HD&UHD sequences of JVET (classes A1, A2 and B) to tune our presets and choose the working points. If you are changing the search algorithm, if you could run it with and without your improvement around the working point you are targeting, that would be amazing (using QP 22, 27, 32 and 37, in accordance with JVET CTC). Otherwise we could also do that, but you'd at least need to explain why you think this would work.

If you are looking into just improving the implementation in a way that does not impact the results, timing of a single encode with and without is enough. We will run it on a bigger set either way.

If you are working on visual quality improvement, you would need to explain a bit what is happening and provide config and sequence that shows the improvement.

If you are working on rate control, we would need to test it internally. We have a somehow versatile test-set including sequences with grain, scene cuts and what-not that we use for that, but because of copyrights cannot share.

If you are correcting a specific issue, just explain the issue with link to the source material and config causing it. We will evaluate the impact on other cases.

[howjmay]

Thanks for the explanation. For my case, I want to replace the current usage of simd_e to sse2neon which is a lighter and single SSE to NEON interpreter. I think it won't affect the behavior at all.
Another question I have right now is I don't know how to run the test. Sorry for that

[adamjw24]

For this change it might be easier to just try and do this for the vvdec project. There you can enable the testsuite (make ... enable-bitstream-download=1 ...), which downloads a lot of test sequences to the ext/bitstreams directory. Since this is purely about runtime, testing on the decoder might be easier (e.g. using the TREE_C_HHI.bit sequence, and since its about SIMD, probably with -t 0), using e.g. the /bin/time utility. Alternatively bitstreams from https://nasext-vaader.insa-rennes.fr/ietr-vaader/OVVC/random_access/ could be used for, testing, since they cover a wider range of tools, thus more of the SIMD would be benchmarked.

[adamjw24]

Btw, before you do the work. Why would you want to use sse2neon instead of simde? Being lightweight is not really so much of an advantage, especially since the latter supports more than the NEON architecture. Do you think sse2neon would be faster? That would be an argument