ci: error out if no riot hashes are found when running test suites

[romainkomorndatadog]

If no riot hashes are found for some reason, scripts/run-test-suite should error and exit.

This also adds a bit of logging for fun.

Example of failure due to no matching hashes:

root@docker-desktop:~/project# scripts/run-test-suite romainlikethelettucewithoutthee
No riot hashes found for pattern: romainlikethelettucewithoutthee

Example of run with matching hashes:

root@docker-desktop:~/project# scripts/run-test-suite ci_visibility
Found 6 riot hashes.
Running riot hash: 12a04f4
INFO:riot.riot:Generating virtual environments for interpreters Interpreter(_hint='3.12')
...

Checklist

• Change(s) are motivated and described in the PR description.
• Testing strategy is described if automated tests are not included in the PR.
• Risk is outlined (performance impact, potential for breakage, maintainability, etc).
• Change is maintainable (easy to change, telemetry, documentation).
• Library release note guidelines are followed. If no release note is required, add label changelog/no-changelog.
• Documentation is included (in-code, generated user docs, public corp docs).
• Backport labels are set (if applicable)

Reviewer Checklist

• Title is accurate.
• No unnecessary changes are introduced.
• Description motivates each change.
• Avoids breaking API changes unless absolutely necessary.
• Testing strategy adequately addresses listed risk(s).
• Change is maintainable (easy to change, telemetry, documentation).
• Release note makes sense to a user of the library.
• Reviewer has explicitly acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment.
• Backport labels are set in a manner that is consistent with the release branch maintenance policy
• If this PR touches code that signs or publishes builds or packages, or handles credentials of any kind, I've requested a review from @DataDog/security-design-and-guidance.
• This PR doesn't touch any of that.

[pr-commenter]

Benchmarks

Benchmark execution time: 2023-09-27 14:53:09

Comparing candidate commit 1c1780b in PR branch romain.komorn/ci_error_if_no_hashes with baseline commit f00acf8 in branch 2.x.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 90 metrics, 0 unstable metrics.

[P403n1x87]

Any value in backporting this?

[romainkomorndatadog]

I'm disabling auto-merge because I don't trust this (because of the ddtracerun test flaking on me).

I'm also actually not sure how many PRs we'll end up blocking if this merges.

@P403n1x87 , as far as backporting this... yeah it's probably be good to backport into 1.20 and 2.0, at least.

[romainkomorndatadog]

Hm, I think this falls down if there are more parallel runs than environments. 6 out of the 8 envs run because we only find 6 hashes.

Put up a fix in my last commit, looks nicer (in that it doesn't crash) but the parallel runs seem a bit pointless:

Run with hashes:

Run that has no hashes:
https://app.circleci.com/pipelines/github/DataDog/dd-trace-py/46740/workflows/0ee091f5-e622-41e8-bdb1-3dee7191a13d/jobs/3036375/parallel-runs/6?filterBy=ALL

[romainkomorndatadog]

https://app.circleci.com/pipelines/github/DataDog/dd-trace-py/46741/workflows/3d298c34-fae0-43af-9c42-ae92f8306fdb/jobs/3036496 looks like it's doing what we want:

[romainkomorndatadog]

@emmettbutler , think this is good to go. I'm a bit hesitant about switching to using an array since we don't really use any in the rest of the script (or in our scripts, usually), but I like being able to give counts for some reason. The other options for counts was something like echo $VAR | wc -w or but that felt "bad" for some reason.