Print list of failed tests at the end of the tornado-tests script #573
Conversation
|
I have an idea. I would like your opinion. Can we add one more line that lists the total number of tests, and any deviations from the current run? I suggest that because currently the |
|
Yes, we need to add if any test/s had |
|
I extended the test script to also report the unit tests that triggered a segfault. Please have another look. |
| @@ -506,21 +509,25 @@ def processStats(out, stats): | |||
| name = name[:-16] | |||
|
|
|||
| if (className + "#" + name in __TORNADO_TESTS_WHITE_LIST__): | |||
| failedTests[className + "#" + name] = "YES" | |||
| print(Colors.RED + "Test: " + className + "#" + name + " in whiteList." + Colors.RESET) | |||
There was a problem hiding this comment.
shall we report the white-listed failed test as orange? in order to make it easy to distinguish, if a non-whitelisted test has failed.
There was a problem hiding this comment.
also, can we also print the list of non-whitelisted tests that failed? And this could be displaying in RED.
I see that if we have white-listed and non-whitelisted tests, we print them all as RED. For example:
In this case, I deliberately made some tests to fail.
|
Here is also the output of |
|
The whitelisted tests are now printed in orange |
|
Is this ready to merge? |
yes |
Improvements ============= - beehive-lab#573: Enhanced output of unit-tests with a summary of pass-rates and fail-rates. - beehive-lab#576: Extended support for 3D matrices. - beehive-lab#580: Extended debug information for execution plans. - beehive-lab#584: Added helper menu for the ``tornado`` launcher script when no arguments are passed. - beehive-lab#589: Enable partial loop unrolling for all backends. - beehive-lab#594: Added RISC-V 64 CPU port support to run OpenCL with vector instructions RVV 1.0 (using the Codeplay OCK Toolkit). - beehive-lab#598: OpenCL low-level buffers tagged as read, write and read/write based on the data dependency analysis. - beehive-lab#601: Feature to select an immutable task graph to execute from a multi-task graph execution plan. Compatibility ============= - beehive-lab#570: Extended timeout for all suite of unit-tests. - beehive-lab#579: Removed legacy JDK 8 and JDK11 build options from the TornadoVM installer. - beehive-lab#582: Restored tornado runner scripts for IntellIJ. - beehive-lab#583: Automatic generation of IDE IntelliJ configuration runner files from the TornadoVM command. - beehive-lab#597: Updated white-list of unit-test and checkstyle improved. Bug Fixes ============= - beehive-lab#571: Fix issues with bracket closing for if/loops conditions. - beehive-lab#572: Fix for printing default execution plans (execution plans with default parameters). - beehive-lab#575: Fix the Level Zero version used for building the SPIR-V backend. - beehive-lab#577: Fix checkstyle. - beehive-lab#587: Fix thread scheduler for new NVIDIA Drivers. - beehive-lab#592: Fix ``Float.POSITIVE_INFINITY`` and ``Float.NEGATIVE_INFINITIVE`` constants for the OpenCL, CUDA and SPIR-V backends. - beehive-lab#596: Fix extra closing bracket during the code-generation for the FPGAs. - Remove the intermediate CUDA pinned memory regions in the JNI code: [link](beehive-lab@9c3f8ce) - Fix bitwise negation operations for the PTX backend: [link](beehive-lab@0db1cd3) - `GetBackendImpl::getAllDevices` thread-safe: [link](beehive-lab@0d44252) - Check size elements for memory segments: [link](beehive-lab@4360385)
Description
This PR extends the
tornado-testsscript to print a summary of the tests that fail along with information about whether they are whilelisted or not. This way it's easy to track the failing tests without having to go through all the output of thetornado-testsscript.Below is the output of the test script with the proposed extension:
Backend/s tested
Mark the backends affected by this PR.
OS tested
Mark the OS where this PR is tested.
Did you check on FPGAs?
If it is applicable, check your changes on FPGAs.
How to test the new patch?
Run
make tests