Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modern Testing #2456

Merged
merged 24 commits into from
Aug 31, 2024
Merged

Modern Testing #2456

merged 24 commits into from
Aug 31, 2024

Conversation

Rot127
Copy link
Collaborator

@Rot127 Rot127 commented Aug 25, 2024

Your checklist for this pull request

  • I've documented or updated the documentation of every API function and struct this PR changes.
  • I've added tests that prove my fix is effective or that my feature works (if possible)

Detailed description

Supersedes #2456

Modernize testing.

The new unit testing is build around yaml files. They can be consumed by the C test engine (cstest) or any bindings easily.

Statistics

The new cstest logs technically two statistics. One by cmocka and one by cstest.
The reason is:

  1. We might want to group tests in the future (by arch or file for example) so we need a global statistics.
  2. Errors of incorrect yaml parsing are not tracked by cmocka.
  3. Also, cmocka doesn't provide the detailed statistics to us. So we cannot print a nice summary at the end.

Changes

  • New cstest C-implementation
  • New py_cstest Python-implementation
  • Moves almost all tests to yaml files.
  • Old tests which can't be moved (because they are effectively integration tests) are kept.
  • Add numerous test jobs in the workflow
  • Adds several CMake test targets
  • Add LoongArch Python bindings
  • Several bug fixes discovered during testing.

Current test coverage of Auto-Sync <ARCH>Mapping.c files by running cstest test/:

file coverage covered
arch/LoongArch/LoongArchMapping.c 67.29% 144 / 214
arch/PowerPC/PPCMapping.c 72.78% 254 / 349
arch/AArch64/AArch64Mapping.c 60.47% 918 / 1518
arch/TriCore/TriCoreMapping.c 45.00% 45 / 100
arch/Alpha/AlphaMapping.c 86.08% 68 / 79
arch/ARM/ARMMapping.c 61.68% 792 / 1284
arch/HPPA/HPPAMapping.c 29.85% 20 / 67

After merge

  • Add windows build to CI

Test plan

Added and all green

Closing issues

closes #1984
closes #2405
closes #2383
closes #2383

- Not removed are the legacy integration tests.
- Detail and asm tests are now in the yaml test files.
The MCUpdater is completely rewritten to generate yaml test files
from the LLVM MC tests.

Also the compatibility header got a test case in CMake.
@Rot127 Rot127 force-pushed the yaml-testing branch 3 times, most recently from c5bd850 to c9c98ad Compare August 26, 2024 11:00
@Rot127 Rot127 force-pushed the yaml-testing branch 3 times, most recently from 864c9dd to d5bdcba Compare August 26, 2024 11:45
@Rot127
Copy link
Collaborator Author

Rot127 commented Aug 28, 2024

@kabeor I fixed the Auto-Sync workflow. Please take another look.

@Rot127 Rot127 mentioned this pull request Aug 28, 2024
2 tasks
We cannot toggle between alias or non aias assembly currently.
So valid tests fail because the non alias asm text is compared.
@Rot127 Rot127 mentioned this pull request Aug 31, 2024
2 tasks
Copy link
Member

@kabeor kabeor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool!

@kabeor kabeor merged commit 191db14 into capstone-engine:next Aug 31, 2024
19 checks passed
@Rot127 Rot127 deleted the yaml-testing branch September 1, 2024 09:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

test_riscv.py failing test_tms320c64x.c does not abort when not enabled Modernize Capstone testing
3 participants