make messages more informative, e.g. vec_grade_vector()

[garrettgman]

I love the swiss army knife approach of vec_grade_vector(), but the messages are not as customized to the student's code as they could be.

Here are some examples

library(tblcheck)
.result <- c("Austin", "Sacramento", "Denver", "Dayona Beach", "Concord")
.solution <- c("Austin", "Sacramento", "Denver", "Daytona Beach", "Concord")
vec_grade_vector()

I'd like something that mentions the students code and points them to what went wrong, e.g: I didn't expect your result to contain the value Dayona Beach.

.result <- c("Austin", "Sacramento", "Denver", "Daytona Beach", "Concord", "Boston")
.solution <- c("Austin", "Sacramento", "Denver", "Daytona Beach", "Concord")
vec_grade_vector()

I'd like to identify the offending value and suggest a remedy: It looks like you added a value that is not in the solution, "Boston". Try removing it.

Why is this a problem?

The messages feel like R error messages. Since they don't talk about the student's code, the student needs to figure out what about their code caused the message. This can be frustrating.

The goal is to make the grading messages always sound like personalized feedback and advice coming from a friendly mentor who has read the student's code. Whenever possible, we should glue in pieces of the student code or result to make it clear what we are saying. And to make it feel personalized.

[rossellhayes]

@garrettgman I like this suggestion a lot! I want to brainstorm a little bit to get your intuition on how this would work.

1. Currently vec_grade_vector returns a message saying "the first n values of your result should be x, y, z ..." if there is an error within the first n values. (n defaults to 3 but can be set by the author.) The less helpful "unexpected values" message comes if there is a problem outside the first n values. Should we replace the whole system, or only improve the case where mistakes occur outside the first n values?
   a. If a student includes multiple unexpected values, should we list multiple of them or only the first mistake?
   b. Do you have an intuition for what we should do if all the values are correct but they are in the wrong order?
2. I like cluing students in to which value is making their vector the wrong length. If the student is missing a value, should we tell them what value it is, or only tell them that they're missing something?
3. If a student has a vector that is both the wrong length and contains incorrect values, which message should be displayed first?
4. The reason the message language is mostly in the passive voice is because vec_grade_vector() may be grading vectors that were the end of a multi-step process rather than being directly created by the student. "Try removing it" is good advice for a lesson teaching c() or :, but might not be helpful if a student doesn't have such direct control over the vector. Is there good phrasing that could cover all situations?
   a. This makes me think it might be useful to add an interface for authors to add their own tag phrases. I'll think about the best way to implement that.

[garrettgman]

1. Should we replace the whole system, or only improve the case where mistakes occur outside the first n values?

Can we reserve showing the first n values as a worst case scenario feedback message? We'd show it if there's evidence that the student misunderstood the prompt, e.g. if the majority of their values are wrong. I remember intending this with my prototype code, but not really achieving it.

2. If the student is missing a value, should we tell them what value it is, or only tell them that they're missing something?

I think we should err on the side of telling them what the missing value is. But if they are missing multiple values, I'd only mention the first. The intent isn't to give away the whole answer, but provide the next nudge to get them back on track. What I feel passionate about here is that the feedback messages always seem like welcome help, and never a second set of checks/error messages to figure out.

3. If a student has a vector that is both the wrong length and contains incorrect values, which message should be displayed first?

This will be more code, but could we have a two stage check? If the length is off by 1-2 we should give them the more informative wrong values message. If the length is off by a lot, we should take that as a sign the student has misunderstood something. We have little hope of guessing what that is, so we would revert to the length message because we cannot be more helpful.

4. Is there good phrasing that could cover all situations?

Great point! I hadn't considered that. Could you give a more concrete example of the case you are describing here?

[rossellhayes]

1. Should we replace the whole system, or only improve the case where mistakes occur outside the first n values?

Can we reserve showing the first n values as a worst case scenario feedback message? We'd show it if there's evidence that the student misunderstood the prompt, e.g. if the majority of their values are wrong. I remember intending this with my prototype code, but not really achieving it.

I think this makes a lot of sense. If you got something totally wrong, this will give you a preview of what you should be aiming for. But if we have more specific feedback, we should focus on that.

2. If the student is missing a value, should we tell them what value it is, or only tell them that they're missing something?

I think we should err on the side of telling them what the missing value is. But if they are missing multiple values, I'd only mention the first. The intent isn't to give away the whole answer, but provide the next nudge to get them back on track. What I feel passionate about here is that the feedback messages always seem like welcome help, and never a second set of checks/error messages to figure out.

Sounds good!

3. If a student has a vector that is both the wrong length and contains incorrect values, which message should be displayed first?

This will be more code, but could we have a two stage check? If the length is off by 1-2 we should give them the more informative wrong values message. If the length is off by a lot, we should take that as a sign the student has misunderstood something. We have little hope of guessing what that is, so we would revert to the length message because we cannot be more helpful.

I like that and I don't think it should be too difficult to implement. I'll get back in touch if it causes any issues, but I think this is a good plan.

4. Is there good phrasing that could cover all situations?

Great point! I hadn't considered that. Could you give a more concrete example of the case you are describing here?

As an example, a student may have unexpected values in a str_subset() exercise if they write a regular expression that is too permissive. In this case, it doesn't feel quite right to say "try removing" the offending value, but something more along the lines of "we didn't expect your result to include x, try adjusting your code so it isn't included". The downside is that any good catch-all is going to sound kind of vague.

[garrettgman]

As an example, a student may have unexpected values in a str_subset() exercise if they write a regular expression that is too permissive. In this case, it doesn't feel quite right to say "try removing" the offending value, but something more along the lines of "we didn't expect your result to include x, try adjusting your code so it isn't included". The downside is that any good catch-all is going to sound kind of vague.

I'm glad you thought of this, and you are absolutely right. I really liked something I saw in another one of your messages today, something like "I didn't expect your result to contain "Boston" and 3 other values." Maybe we should use a similar message here. Balancing general/specific feedback seems like a tough problem to solve. What do you think of a two pronged approach:

1. Wherever we can, we attempt to glue something from the result into the feedback message to make it seem more personalized
2. We encourage authors to use a check function before calling vec_grade_vector() when they want to "mask" a vec_grade_vector() message with a more specific message, e.g.

vec_fail_if_extra_values("It looks like you added a value that is not in the solution, {.extra[1]}. Try removing it.")
vec_grade_vector()

^^^Not much thought went into this. So the details are probably flaky. But I like the idea that vec_fail_if_extra_values() knows that I'm using it to write a custom message and that I'll want to mention something obvious, the extra vaues, which it makes available for gluing into my message.