How do we express the intention of non-obvious test cases in our canonical-data.json files?

[petertseng]

I'm spinning this off from #230 since #230 forked off into at least three separate topics.

The problem statement I wish to discuss:

When I look at a canonical-data.json, how can I be sure that every test case is necessary? For each test case:

• What does it test for?
• What mistaken solutions might it help catch?

If it is not already clear from the test description, I want some way to tell.

From #230, we have some suggestions from @kotp and @kytrinyx that a "notes" key for the non-obvious cases can be used. I have similarly used a "#" key (why "#"? Probably because it starts a comment line in some language, but I think I prefer a descriptive word than "#") such as at https://github.com/exercism/x-common/blob/master/exercises/largest-series-product/canonical-data.json#L89 . Note that that specific "#" was to explain why the test case is correct rather than why it is necessary, so it's slightly different, but I imagine it could still be changed to "notes".

One possible solution I anticipate people saying is "let's just use Git commit messages". I had some thought that it might take more time to look through those than to have it right there in the file, but I'm open to discussion on that one.

On the other hand, if we decide to use "notes", then we would want to update https://github.com/exercism/x-common#test-data-format-canonical-datajson .

[petertseng]

From my reading of #382 (comment)

This also feels different from canonical-data.json as well, as that feels like it is the end result of the discussion, and most people who want to use it to generate a new implementation should be able to follow it without worrying about the details, unless they're interested in them.

This may imply that a "notes" key is too much information for the canonical-data.json file, and that the answers to the questions I pose might be better served by having a separate file and/or looking through the commit messages.

I think this basically comes down to different expectations of canonical-data.json. The quoted comment implies that someone who is porting the exercise to a track has the idea "don't give me the details of why this test is there, I just care about what the tests are", whereas someone who asks the above questions is approaching from the angle of maintaining the JSON file and trying to bring it closer to perfection (as a known quote says, achieved when there is nothing more that can be taken away!)

How can we serve these two different audiences?

[IanWhitney]

One possible solution I anticipate people saying is "let's just use Git commit messages".

This is my answer. Git commit messages are for the "Why". Trying to convert a structured format (like json) so that it contains all the context sounds like trying to play basketball with a hockey puck. You might be able to make it work, but it's probably not going to be much fun.

[petertseng]

On a basic level, I realize an analogy for this issue is like the difference between putting rationale for a certain piece of code in a code comment versus putting it a commit message. When is the right time to do one versus the other? This is something I don't have a good answer for personally. I would be interested in whether anyone has rules they use for deciding this. I'll do some looking and report back too.

As it relates to action on this issue: My need for something other than Git commit messages would be more convincing if I can point to an example of an exercise in which

1. I was confused by a test case, and
2. Git commit messages were not sufficient for me to understand.

I have no current example fitting these criteria, so that makes me unsure whether I should pursue this.

I think we have been good at asking for rationale in commit messages. And since people are already going to put their rationale in commit messages. One could argue that it is duplication to put the same rationale in both commit messages and somewhere in the JSON file!

If I can't come up with a good example for when Git messages did not serve me, I should close this issue, once I confirm that we have some document with guidelines for commit messages (I think we have one, but I want to be sure).

[kytrinyx]

One possible solution I anticipate people saying is "let's just use Git commit messages".

I think git commit messages are for the why, but for a slightly different purpose. Git commit messages are something that you would dig into only if you wanted to change something and were unsure about whether or not something has been discussed before. In this case I think that we'd be interested in surfacing the why on a more broader scale: anyone implementing the exercise should be able to understand what philosophical vectors we've traveled along and why we ended up where we did. That seems like something worth surfacing in a policy document or a "notes" section in the canonical data.

I like the distinction that @petertseng makes between # for what and notes for why for a specific test case. That said, I think that since the why is not ever intended to be machine readable, and it's also likely that it doesn't apply to specific tests, a separate document would be easier to use.

[petertseng]

That seems like something worth surfacing in a policy document or a "notes" section in the canonical data.

If we pick the former, it sounds like something #382 would handle.

(Just trying to figure out if I should still keep this issue open)

[petertseng]

I should close this issue, once I confirm that we have some document with guidelines for commit messages

Confirm: https://github.com/exercism/x-common/blob/master/CONTRIBUTING.md#commit-messages

That's the last thing I had to check before closing the issue - the policy document discussion can go in #382, I think?