Fix broken sample and add test for sample

[nnegrey]

No description provided.

[kurtisvg]

It might be worthwhile to update these tests to check that some labels were successfully returned, instead of a specific label (unless the docs specifically mention it should be detecting that label)

[nnegrey]

Like switching the functions to return the result and then check if the size of returned labels are not 0?

I know that for ML APIs, I believe we still want to validate some of the results to make sure the results are accurate. But let me go double check.

[kurtisvg]

Just something to consider if the labels change a lot. Not blocking PR.

[nnegrey]

Merging for now, will check with team on testing changes.

[beccasaurus]

+1 @kurtisvg Love the idea!

This came up in conversation only last week for testing responses from Machine Learning APIs based on models that can and do change over time

I love the idea of providing a list of test inputs/outputs and verifying that at least 1 of the assertions passed

It is very much à la Data Driven Tests aka Parameterized Tests

(see: JUnit4 Theories & other xUnit-style frameworks which implement 'Theories' for data-driven tests – C# xUnit Theory, Go Table-Driven-Tests, Python parameterized package, ...)

The main difference versus typical parameterized tests is that they usually exist to verify ALL SCENARIOS SUCCEED (versus any one of many scenarios succeed).

---

pseudocode example of parameterized tests

method testMySample
  run_sample "input.png"

  assert output includes "Label: Dog"
end

^--- rather than 1 test that asserts "Dog" is returned as a label when analyzing an image file, provide a LIST of input/outputs to test

test_cases = [
  [ "dog.png", "Label: Dog" ],
  [ "cat.png", "Label: Cat" ],
  [ "cat.png", "Label: Feline" ]
]

method data_driven_test file_name, output_assertion
    run_sample file_name

  assert output includes output_assertion
end

And we want to assert at least one of the test cases passes

---

I love this. But I also want to be careful implementing it at this time. Because we have test-case consistency across 7 languages. When we start using this technique, I want to be clear about the expectations across languages. This also adds a new responsibility to the author of the canonical sample: rather than coming up with 1 test case, they will be tasked with coming up with multiple which should be stable over time – these should probably come from the product team.

#misc: I'm in the design phases of a generator for code sample tests (initially only for code samples that are generated as well) and this is one of my design requirements. I am backing in table/data-driven tests with the toggle option of either verifying that ALL SUCCESS (default) or ANY ONE SUCCEEDS (available option)