[AzureDevOpsTests] Refactor unit tests to improve stability

[mehmetseckin]

As discussed in #2412, a failed build yielded a no tests badge. This is a valid scenario, but the unit tests that assert the badge message for different configurations were failing as they assumed a successful response that contains tests for live requests.

This PR adds a mock response and corrects the RegEx logic used in the unit tests to mitigate this scenario and decouple from the live API.

/cc @paulmelnikow, @calebcartwright

[shields-ci]

	Messages
📖	✨ Thanks for your contribution to Shields, @mehmetseckin!

Generated by 🚫 dangerJS

[paulmelnikow]

Hi! Thanks so much for responding to this.

The reason isBuildStatus excludes "no tests" is because for most APIs, a branch with tests should never return "no tests". If it does, it's probably a problem in our error-handling code or because something has changed on the server.

It's fine to add mock tests, though they should be in addition to the live tests. To fix the live tests, I'd suggest adding a new Joi matcher in this test file that includes "no tests" along with the other possible results.

The reason we generally prefer live tests is because server APIs change and server behaviors can change in subtle ways we depend on. This way we often find out about the failures first. Sometimes it's the only way we find out at all, because issues aren't always reported. Yes, they're more work to maintain, though we get enough value out of them that it's worth it… and we're continually getting better at avoiding false positives. There's some background discussion on the strategy in #927.

You mentioned in #2411 this was an undocumented API, so all the more reason to have our tests fail if something changes upstream…

[mehmetseckin]

Hi @paulmelnikow! Thanks a lot for the detailed review, it now makes sense to keep an eye on the public APIs through unit tests.

I switched the mocked tests to be asserted by expected preset values, and added live tests that are asserted dynamically using regex. I added no tests as an acceptable result for live tests.

Thanks again!

[paulmelnikow]

Thanks! This looks good, just one comment to avoid some repetition

[paulmelnikow]

Lovely, thank you.

[shields-deployment]

This pull request was merged to master branch. This change is now waiting for deployment, which will usually happen within a few days. Stay tuned by joining our #ops channel on Discord!

After deployment, changes are copied to gh-pages branch: