Is there a way to measure performance or is there a benchmark for liboqs

[gabrofig]

Hi all,
As title says, is it possible to measure the performance in time of execution and memory usage (for example) of liboqs?

Thank you!

[SWilson4]

Hi @gabrofig, liboqs has a few test programs for measuring algorithm performance. See section 3 of the quickstart guide for details.

[gabrofig]

Thank you Michael. How could I run the test_kem_mem.c for example? I am trying to run directly vai terminal but it is not working.
Apologies for the noob question, but I must be doing something wrong.

[baentsch]

If memory serves, the program gives Usage help (if correctly built). Pretty similar to the other test programs (algname) and probably a test stage. Anyway, please always provide command tried, error message and what you expect.

[gabrofig]

I tried running test_mem.py directly in Visual Studio Code and got the following message

kem_name ML-KEM-1024
running subprocess
. > /home/g339268/liboqs 2/liboqs/build/tests/test_kem_mem ML-KEM-1024 0
command: ['/home/g339268/liboqs 2/liboqs/build/tests/test_kem_mem', 'ML-KEM-1024', '0']
result ------------------------------------- CompletedProcess(args=['/home/g339268/liboqs 2/liboqs/build/tests/test_kem_mem', 'ML-KEM-1024', '0'], returncode=0, stdout=b'Configuration info\n==================\nTarget platform: x86_64-Linux-6.5.0-35-generic\nCompiler: gcc (11.4.0)\nCompile options: [-Wa,--noexecstack;-O3;-fomit-frame-pointer;-fdata-sections;-ffunction-sections;-Wl,--gc-sections;-Wbad-function-cast]\nOQS version: 0.10.1-dev\nGit commit: 6ee5de2\nOpenSSL enabled: Yes (OpenSSL 3.0.2 15 Mar 2022)\nAES: NI\nSHA-2: OpenSSL\nSHA-3: C\nOQS build flags: BUILD_SHARED_LIBS OQS_DIST_BUILD OQS_OPT_TARGET=generic CMAKE_BUILD_TYPE=Release \nCPU exts active: AES AVX AVX2 BMI1 BMI2 PCLMULQDQ POPCNT SSE SSE2 SSE3\n================================================================================\nExecuting keygen for KEM ML-KEM-1024\n================================================================================\nStarting OQS_MEM_SECURE_FREE 1')
kem_name ML-KEM-1024
running subprocess
. > /home/g339268/liboqs 2/liboqs/build/tests/test_kem_mem ML-KEM-1024 1
command: ['/home/g339268/liboqs 2/liboqs/build/tests/test_kem_mem', 'ML-KEM-1024', '1']
result ------------------------------------- CompletedProcess(args=['/home/g339268/liboqs 2/liboqs/build/tests/test_kem_mem', 'ML-KEM-1024', '1'], returncode=0, stdout=b'Configuration info\n==================\nTarget platform: x86_64-Linux-6.5.0-35-generic\nCompiler: gcc (11.4.0)\nCompile options: [-Wa,--noexecstack;-O3;-fomit-frame-pointer;-fdata-sections;-ffunction-sections;-Wl,--gc-sections;-Wbad-function-cast]\nOQS version: 0.10.1-dev\nGit commit: 6ee5de2\nOpenSSL enabled: Yes (OpenSSL 3.0.2 15 Mar 2022)\nAES: NI\nSHA-2: OpenSSL\nSHA-3: C\nOQS build flags: BUILD_SHARED_LIBS OQS_DIST_BUILD OQS_OPT_TARGET=generic CMAKE_BUILD_TYPE=Release \nCPU exts active: AES AVX AVX2 BMI1 BMI2 PCLMULQDQ POPCNT SSE SSE2 SSE3\nttttt================================================================================\nExecuting encaps for KEM ML-KEM-1024\nttttt================================================================================\nStarting OQS_MEM_SECURE_FREE 1')
kem_name ML-KEM-1024
running subprocess
. > /home/g339268/liboqs 2/liboqs/build/tests/test_kem_mem ML-KEM-1024 2
command: ['/home/g339268/liboqs 2/liboqs/build/tests/test_kem_mem', 'ML-KEM-1024', '2']
result ------------------------------------- CompletedProcess(args=['/home/g339268/liboqs 2/liboqs/build/tests/test_kem_mem', 'ML-KEM-1024', '2'], returncode=0, stdout=b'Configuration info\n==================\nTarget platform: x86_64-Linux-6.5.0-35-generic\nCompiler: gcc (11.4.0)\nCompile options: [-Wa,--noexecstack;-O3;-fomit-frame-pointer;-fdata-sections;-ffunction-sections;-Wl,--gc-sections;-Wbad-function-cast]\nOQS version: 0.10.1-dev\nGit commit: 6ee5de2\nOpenSSL enabled: Yes (OpenSSL 3.0.2 15 Mar 2022)\nAES: NI\nSHA-2: OpenSSL\nSHA-3: C\nOQS build flags: BUILD_SHARED_LIBS OQS_DIST_BUILD OQS_OPT_TARGET=generic CMAKE_BUILD_TYPE=Release \nCPU exts active: AES AVX AVX2 BMI1 BMI2 PCLMULQDQ POPCNT SSE SSE2 SSE3\n================================================================================\nExecuting decaps for KEM ML-KEM-1024\n================================================================================\nshared secrets are equal\n0000000000000000000test 1Starting OQS_MEM_SECURE_FREE 1')
======================================= test session starts =======================================
platform linux -- Python 3.10.12, pytest-6.2.5, py-1.10.0, pluggy-0.13.0
rootdir: /home/g339268/liboqs 2/liboqs/tests
plugins: xdist-2.5.0, forked-1.4.0
collected 32 items

test_mem.py ................................ [100%]

======================================= 32 passed in 2.90s ========================================

From what I understood, this result confirms that KEM was executed successfully. However, I would like to see the results for the memory usage or time of execution of each steps (keygen, encap, decap) for example.

[baentsch]

However, I would like to see the results for the memory usage or time of execution of each steps (keygen, encap, decap) for example.

The OQS test code is there to be used within suitable valgrind executions (tuned to what your particular interest is, stack, heap, whatever). I'm afraid there are no ready-made invocations for simple use (just for constant time code path testing). If you set up something that generates data of interest and value, please consider a PR @gabrofig.

[baentsch]

PS: Just remembered I did this many years ago: Take a look at https://github.com/open-quantum-safe/profiling/blob/main/perf/scripts/run_mem.py

[baentsch]

I'm sorry, I can't really help here as I don't have my computer around this week. Maybe @SWilson4 can help now. Otherwise I try to look into this next week. ⁣BlueMail for Android herunterladen ​ Am 13. Aug. 2024, 14:57, um 14:57, gabrofig ***@***.***> schrieb:

…

Thank you Michael,  As I was running the command docker buildx build -t oqs-perf:arm64 --platform=linux/arm64 . to build ARM64, I believe I faced a restriction for permission in the docker. Is there a way to use only the run_mem.py script that you shared with me to evaluate KEM performance locally? If yes, do I need to modificate it? -- Reply to this email directly or view it on GitHub: #1810 (comment) You are receiving this because you commented. Message ID: ***@***.***>

[gabrofig]

Thank you Michael, I will kindly ask Spencer in case he can support.
When I tried to build ARM64, with the docker buildx build -t oqs-perf:arm64 --platform=linux/arm64 . I got the following errors that I assume are due lack of permission to the docker (?)

Is this build necessary to use the run_mem.py script for KEM? If not, how can I use this script to assess performance for KEM?
I appreciate any feedback!

[gabrofig]

Just adding: I simply ran sudo docker run -v /home/tests:/opt/test/results oqs-perf and got some results as bellow before having the following error: Trying to mount S3...
./mount_s3.sh expects BUCKETSECRETS to authenticate to S3. Exiting without mount.
Couldn't mount S3 bucket. Not copying out test results.

I believe those results are the performance ones that I mentioned, correct?

[baentsch]

I believe those results are the performance ones that I mentioned, correct?

I'm not sure I understand what/where you mentioned them (?)

Couldn't mount S3 bucket. Not copying out test results.

That error is expected as you probably didn't pass the required access credentials for some S3 bucket where to store the results (and surely don't have the ones OQS uses :)

[gabrofig]

Thank you Michael.
I mean that, for example KEM 1024, the command docker run -v /home/tests:/opt/test/results oqs-perf generated results for total time (s)m time (us): mean, pop stdev CPU cycles etc. I believe those are already at least some of the metrics for the KEM algorithm. I am only missing memory consumption in those tests. Is there a way to run only a test to measure the memory consumption?

Kind regards,

[baentsch]

Is there a way to run only a test to measure the memory consumption?

Yes. Please dissect the core test runtime script.

[gabrofig]

I Think I found it. I ran locally ./run-tests.sh and after some path manipulation and printing partial results for test_kem_mem, I got the following results

Thanks a lot for all the support from you guys!

[baentsch]

UW. OK to mark any comment as answer and close the issue this way?

[gabrofig]

Done, thank you again!