[CI] Deduplicate and clean XML test reports #12332
Conversation
gigiblender
force-pushed
the
tests-report
branch
from
7329e6f to
e3bf8de
Compare
gigiblender
force-pushed
the
tests-report
branch
from
e3bf8de to
c229e2b
Compare
| ) | ||
| ) | ||
| if item_shard_index != shard_index: | ||
| items.remove(item) |
There was a problem hiding this comment.
The only hesitation I have with this is that now if someone goes and looks for a test named test_abc in one of the 10 CPU integration shard logs, they'd have to search through every log until they find it and more importantly know that it's not weird that the test doesn't show up at all in most of the logs. Do you have any ideas on how we could make this a better ux?
There was a problem hiding this comment.
I am not very sure how we could make this a better UX.
I understand your concern, but I'm wondering how often someone does a manual search for a specific test through the shard logs. If a test fails, the corresponding shard will appear red in the UI.
Alternatively, we can keep the old log structure (showing the skipped tests), and I can filter them later when processing the XML logs.
There was a problem hiding this comment.
I see what you mean, I suppose it would be pretty rare. In a follow up or something we can add a message that links to the log where all the steps are catted together
Looks like #12332 broke [the CI](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/3992/pipeline/389) due to a misuse in the `aws s3 cp` command. This PR fixes that.
This PR does the following: - When running on a shard, removes the tests that will not run on the current shard from the suite. We'll no longer get the message `Test running on shard X of Y` in the XML reports. This will result in cleaner test reports. - Adds the shard index or `no-shard` to the generated XML report file name. Currently, the reports generated when sharding is present are overwritten in S3 by the last shard to finish in the CI pipeline. This change is needed as part of apache#11670. - Since the same tests might run on different configurations (CPU, GPU), it uploads the result of each configuration in a subdirectory in S3 (e.g `/pytest-results/frontend_aarch64`).
Looks like apache#12332 broke [the CI](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/3992/pipeline/389) due to a misuse in the `aws s3 cp` command. This PR fixes that.
Looks like apache#12332 broke [the CI](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/3992/pipeline/389) due to a misuse in the `aws s3 cp` command. This PR fixes that.
This PR does the following:
Test running on shard X of Yin the XML reports. This will result in cleaner test reports.no-shardto the generated XML report file name. Currently, the reports generated when sharding is present are overwritten in S3 by the last shard to finish in the CI pipeline. This change is needed as part of [CI Problem] Figure out a way to assert certain tests aren't skipped in CI #11670./pytest-results/frontend_aarch64).cc @Mousius @areusch @driazati