Added score calculations #3040

TheRealJessicaLi · 2021-04-04T02:07:30Z

Explanation

Added score calculations (with testing) for practice sessions. Mastery calculations to be implemented in a separate PR.

Checklist

The PR title starts with "Fix #bugnum: ", followed by a short, clear summary of the changes. (If this PR fixes part of an issue, prefix the title with "Fix part of #bugnum: ...".)
The PR explanation includes the words "Fixes #bugnum: ..." (or "Fixes part of #bugnum" if the PR only partially fixes an issue).
The PR follows the style guide.
The PR does not contain any unnecessary auto-generated code from Android Studio.
The PR is made from a branch that's not called "develop".
The PR is made from a branch that is up-to-date with "develop".
The PR's branch is based on "develop" and not on any other branch.
The PR is assigned to an appropriate reviewer in both the Assignees and the Reviewers sections.

linted again more linting :) linted

BenHenning · 2021-04-06T02:10:04Z

@TheRealJessicaLi looking at the CI results, it seems there might be some issues with building & running the domain questions tests. Do they build & run for you locally?

Also, I'll need to take a closer look at this tomorrow or Wendesday--apologies for the delay.

vinitamurthi · 2021-04-06T10:48:53Z

domain/src/main/java/org/oppia/android/domain/question/QuestionAssessmentCalculation.kt

+    for (questionMetric in questionSessionMetrics) {
+      val totalHintsPenalty = questionMetric.numberOfHintsUsed * viewHintPenalty
+      val totalWrongAnswerPenalty =
+        (questionMetric.numberOfAnswersSubmitted - 1) * wrongAnswerPenalty


Do we need to ensure that this isn’t a negative value (I.e. numberOfAnswersSubmitted has to be greater than 0?

I looked into QuestionAssessmentProgressController and it seems like we're only marking a question as completed once the user actually submits an answer for that question. Since we're not calculating scores until all the questions are completed, I don't think numberOfAnswersSubmitted would ever be 0.

Might it be worth filtering the metrics on those that have at least 1 submitted answer? That might be simple with Kotlin:

for (questionMetric in questionSessionMetrics.filter { it.numberOfAnswersSubmitted > 0 }) {

I don't think we'd want to filter those out because we would still want to include those questions in the total score. As an alternative, I could make sure the computed totalWrongAnswerPenalty is non-negative?

vinitamurthi · 2021-04-06T10:50:35Z

domain/src/main/java/org/oppia/android/domain/question/QuestionAssessmentCalculation.kt

+      this.totalScore.numerator += questionScore
+      for (linkedSkillId in questionMetric.question.linkedSkillIdsList) {
+        if (!scorePerSkillMapping.containsKey(linkedSkillId)) continue
+        scorePerSkillMapping[linkedSkillId]!!.numerator += questionScore


There could be cases where the score per skill mapping has a numerator and a denominator as 0. Are we handling that case anywhere? Ideally it’s best to remove those entries lower in the stack so that when we actually divide the numerator and denominator we do not need to handle that scenario

Addressed! Thanks for catching that

Maybe I missed this..won't we still have cases where both the numerator and denominator can be 0?

We're removing any skills with a 0 numerator/denominator in the computeAll() method

vinitamurthi · 2021-04-06T10:51:31Z

domain/src/main/java/org/oppia/android/domain/question/QuestionAssessmentCalculation.kt

+  /** Compute the overall score as well as the score and mastery per skill. */
+  internal fun computeAll(): UserAssessmentPerformance {
+    calculateScores()
+    // TODO: set up mastery calculations


Can you create an issue for this and add the issue number here? I.e. TODO(#issuenum)....

domain/src/main/java/org/oppia/android/domain/question/QuestionAssessmentCalculation.kt

.../src/test/java/org/oppia/android/domain/question/QuestionAssessmentProgressControllerTest.kt

BenHenning · 2021-04-08T03:07:57Z

Will take a pass after the latest changes are addressed.

linted more linting

TheRealJessicaLi · 2021-04-09T19:24:08Z

Will take a pass after the latest changes are addressed.

Just addressed Vinita's comments!

BenHenning · 2021-04-10T01:23:13Z

Sorry, will need to look at this tomorrow (but maybe Monday).

BenHenning

Thanks @TheRealJessicaLi! Sorry for the super long delay here. Took a first pass (mostly just considering some of the high-level decisions before doing a closer review).

BenHenning · 2021-04-13T03:11:55Z

domain/src/main/java/org/oppia/android/domain/question/QuestionAssessmentCalculation.kt

+/**
+ * Private mutable class that stores a grade as a fraction.
+ */
+private class MutableFractionGrade(var numerator: BigDecimal, var denominator: BigDecimal)


At a high-level, why are we using BigDecimal? It seems like a pretty heavy weight solution, and I don't see the need here for perfect precision. What are we trying to mitigate with it?

This was to avoid floating point precision errors when doing computations, but I just saw your comment about using integers so I'll try that instead

BenHenning · 2021-04-13T03:12:50Z

domain/src/main/java/org/oppia/android/domain/question/QuestionAssessmentCalculation.kt

+  /** Compute the overall score as well as the score and mastery per skill. */
+  internal fun computeAll(): UserAssessmentPerformance {
+    calculateScores()
+    // TODO(#3067): create mastery calculations method


Nit (for consistency):

Suggested change

// TODO(#3067): create mastery calculations method

// TODO(#3067): Create mastery calculations method.

BenHenning · 2021-04-13T03:15:03Z

domain/src/main/java/org/oppia/android/domain/question/QuestionConstantsProvider.kt

+
+  @Provides
+  @WrongAnswerPenalty
+  fun provideWrongAnswerPenalty(): Double = 0.1


For this & above: it feels a bit like the unit is missing here. Maybe 'WrongAnswerScorePenalty' would be clearer?

BenHenning · 2021-04-13T03:15:47Z

model/src/main/proto/question.proto

@@ -82,14 +82,20 @@ message AnsweredQuestionOutcome {
  bool is_correct_answer = 2;
 }

+// A user's grade for a practice session represented as a fraction.
+message FractionGrade {


This is a bit confusing since fractions can't be a real value divided by a real value. For what reason do we want to keep a numerator and denominator here?

I believe Chantel wants to display the score for each skill as a fraction (so we need both numerator and denominator), and Oppia web is calculating scores as real values instead of integers, so I did the same for consistency. Do we want to multiply everything by a factor of 10 so we have integers instead?

I think that's a worthwhile simplification. I see the current implementation and I find it much easier to follow without the floating point bits.

That being said, a few thoughts:

If you're multiplying everything by '10', we should make that '10' a Dagger constant since it's now a magic value (e.g. something like 'max points per question' or reuse the existing constant)

'numerator' and 'denominator' are still really unclear. Could we just use the same terms that are being used elsewhere? E.g. 'pointsReceived' & 'totalPointsAvailable', respectively?

Done, thanks!

Following up on this--can these values be ints now that everything is being scaled? You could always perform the final decimal calculation in the UI and keep things ints at the domain layer to simplify management.

I kept it as a double because I think Chantel may want to display the scores as a fraction in the UI (e.g. "you got score x / y"), so in that case I thought it would be better to calculate the final values to be consistent with the way Oppia web does it (which may not be integers). To do the calculation, we would also need access to the multiplier constant, which is in the domain module. I thought it made more sense to complete the calculations here before sending off the final score (ready for display) to the app module. Let me know what you think!

BenHenning · 2021-04-13T03:16:33Z

domain/src/main/java/org/oppia/android/domain/question/QuestionConstantsProvider.kt

+
+  @Provides
+  @ViewHintPenalty
+  fun provideViewHintPenalty(): Double = 0.1


Out of curiosity, do these need to be represented as doubles? We could alternatively employ a point system wherein max score is 10 points and penalties are 1 point each. That might let us keep integers everywhere along the computation path which would offer perfect precision, too, until the final division. Would that work?

That should work and it makes sense. I believe we went with a decimal system for web and so thats why we translated the same thing here

I fixed the implementation to use integers in the score calculation step under the hood (by multiplying everything by a factor of 10), but I'm returning doubles as the result (by dividing everything by 10 again) because Chantel said we may be displaying the scores in fraction form and I wanted to make sure the scores match the decimal system used for web in that case. Let me know what you think!

vinitamurthi · 2021-04-13T03:54:58Z

domain/src/main/java/org/oppia/android/domain/question/QuestionAssessmentCalculation.kt

+      }
+    }
+
+    // TODO(#3067): set up finalMasteryPerSkillMapping


Suggested change

// TODO(#3067): set up finalMasteryPerSkillMapping

// TODO(#3067): Set up finalMasteryPerSkillMapping

vinitamurthi · 2021-04-13T03:55:45Z

domain/src/main/java/org/oppia/android/domain/question/QuestionConstantsProvider.kt

+
+  @Provides
+  @ViewHintPenalty
+  fun provideViewHintPenalty(): Double = 0.1


That should work and it makes sense. I believe we went with a decimal system for web and so thats why we translated the same thing here

vinitamurthi · 2021-04-13T03:57:47Z

domain/src/main/java/org/oppia/android/domain/question/QuestionAssessmentCalculation.kt

+      this.totalScore.numerator += questionScore
+      for (linkedSkillId in questionMetric.question.linkedSkillIdsList) {
+        if (!scorePerSkillMapping.containsKey(linkedSkillId)) continue
+        scorePerSkillMapping[linkedSkillId]!!.numerator += questionScore


Maybe I missed this..won't we still have cases where both the numerator and denominator can be 0?

BenHenning

I spent a bit longer on trying to simplify the calculation class that I intended. LMK what you think about the suggestion and then I will take a pass on the other parts of the PR.

BenHenning · 2021-05-01T01:03:59Z

domain/src/main/java/org/oppia/android/domain/question/QuestionAssessmentCalculation.kt

+  private val scorePerSkillMapping: MutableMap<String, MutableFractionGrade>
+) {
+  /** Calculate the user's overall score and score per skill for this practice session. */
+  private fun calculateScores() {


I think there are generally two expected ways to approach this problem (unless I'm missing some fundamental requirements):

Each time a question is answered, determine what needs to be tracked & track that. Results are a filtered aggregate/combination of this data.

Each time a question is answered, we record that it was answered. When results are needed, we calculate everything in that moment.

You're right that mutable state is needed now, but I think the choices allow flexibility in storing the mutable state either at the stats level (option 1) or at the event level (option 2). Option 1 is more performant when aggregating since we do some work ahead of time, and option 2 is more performant when recording. We should pick whichever option reflects typical access patterns, and which simplifies the implementation.

I think the current approach is more of option 2 in aggregation (everything is happening in compute) but using the storage of option 1. I don't quite see why we need to store MutableFractionGrade when it's only being changed/used in the compute function. It seems a bit like it's memoization, but unless compute is being called a lot that probably isn't needed.

Now, stepping back a moment. Given that the class no longer is storing state in Dagger (& thus has nothing to reset across sessions--we can just create a new one), and it's intended that computeAll only be computed at the end of the session, this additional caching seems a lot less necessary. Avoiding it actually results in a lot of simplifications, I think. One alternative that follows this pattern might be something like (which also conveniently avoids mutable state):

fun computePerformance(questionSessionMetrics: List<QuestionSessionMetrics>): UserAssessmentPerformance { val allQuestionScores: List<Int> = questionSessionMetrics.map { it.computeQuestionScore() } val totalPointsScored = allQuestionScores.sum() val totalAvailablePoints = allQuestionScores.size * maxScorePerQuestion // Note to Jessica: this also avoids needing to pass in the list to the class's constructor since we no longer need to initialize the list ahead of time. I think that results in a simpler API even though it comes at the expense of calling computeQuestionScore multiple times (thought that *could* be memoized if we found it to have performance issues). val scoresPerSkill: Map<String, List<Int>> = questionSessionMetrics.flatMap { questionMetric -> // Convert to List<Pair<String, QuestionSessionMetrics>>> questionMetric.linkedSkillIdsList.map { skillId -> skillId to questionMetric} }.groupBy( // Covert to Map<String, List<QuestionSessionMetrics>> for each linked skill ID. { (skillId, _) -> skillId }, valueTransform = { (_, questionMetric) -> questionMetric } ).mapValues { (_, questionMetrics) -> // Compute the question score for each question, converting type to Map<String, List<Int>> questionMetrics.map { it.computeQuestionScore() } } // Assemble & return the proto, plus any other needed bookkeeping. Use the same sum()/size() * max to compute the total scores/available for each question metric. } private fun UserAssessmentPerformance.computeQuestionScore(): Int = if (!didViewSolution) { val hintsPenalty = numberOfHintsUsed * viewHintPenalty val wrongAnswerPenalty = getAdjustedWrongAnswerCount() * wrongAnswerPenalty (maxScorePerQuestion - hintsPenalty - wrongAnswerPenalty).coerceAtLeast(0) } else 0 private fun UserAssessmentPerformance.getAdjustedWrongAnswerCount(): Int = (UserAssessmentPerformance - 1).coerceAtLeast(0)

(this is not complete & could be further split up if desired). https://stackoverflow.com/a/43614691 was a big help for the grouping part since I couldn't quite figure out how to map against the linked skill IDs.

WDYT?

TheRealJessicaLi · 2021-05-01T19:27:26Z

I spent a bit longer on trying to simplify the calculation class that I intended. LMK what you think about the suggestion and then I will take a pass on the other parts of the PR.

I took a look at the suggestions you made and I agree that it makes things a lot cleaner and safer - I've updated QuestionAssessmentCalculation to reflect the changes. Thanks for taking the time to flesh out the pattern! I'll try to use something similar when I'm implementing mastery calculations to avoid mutable state as much as possible.

BenHenning · 2021-05-06T09:13:54Z

Sorry, been really busy with non-work items this week. Will need to take a look later in the week.

vinitamurthi

LGTM!

vinitamurthi · 2021-05-07T11:08:15Z

domain/src/main/java/org/oppia/android/domain/question/QuestionAssessmentCalculation.kt

+      pointsReceived = allQuestionScores.sum().toDouble() / internalScoreMultiplyFactor
+      totalPointsAvailable =
+        allQuestionScores.size.toDouble() * maxScorePerQuestion / internalScoreMultiplyFactor


Do we need to divide both the numerator and denominator of the FractionGrade by the factor? Wont the fraction stay equivalent regardless?

It might also be simpler to just document the multiplier in the other constants rather than factoring it in with math (since it's unlikely we need to tweak that without changing the other constants).

@vinita @BenHenning My reasoning for dividing both the numerator and denominator by the multiplier was to make sure the final FractionGrade's numerator and denominator are consistent with the values that we would have gotten on Oppia web. Since Chantel said she may want to display the score as a fraction (e.g. "you got x / y points"), I thought it would be better to do that here so the app layer doesn't have to simplify it.

I made a separate constant for this multiplier because it's possible that the maxScorePerQuestionConstant can be different from the multiplier if the scoring system were to the change in the future. As an example right now, for mastery calculations, maxMasteryGainPerQuestion is 10, but we multiplied all the constants by a factor of 100 to maintain integer values. In the case that they are different (like for masteries), we would need to use both values to calculate the final scores/masteries. I thought keeping this multiplier separate would be good for consistency and also leave room for future changes so the math still checks out.

Let me know what you think, thanks!

vinitamurthi · 2021-05-07T11:09:29Z

domain/src/main/java/org/oppia/android/domain/question/QuestionAssessmentCalculation.kt

+        pointsReceived = scores.sum().toDouble() / internalScoreMultiplyFactor
+        totalPointsAvailable =
+          scores.size.toDouble() * maxScorePerQuestion / internalScoreMultiplyFactor


Same comment here

See comment above!

BenHenning

Thanks @TheRealJessicaLi. Took a full pass on the PR & have a few comments. Note that I'm resetting my previous review status to unblock this PR since I'll be largely unavailable over the next couple weeks.

BenHenning · 2021-05-08T01:55:52Z

domain/src/main/java/org/oppia/android/domain/question/QuestionAssessmentCalculation.kt

+      pointsReceived = allQuestionScores.sum().toDouble() / internalScoreMultiplyFactor
+      totalPointsAvailable =
+        allQuestionScores.size.toDouble() * maxScorePerQuestion / internalScoreMultiplyFactor


It might also be simpler to just document the multiplier in the other constants rather than factoring it in with math (since it's unlikely we need to tweak that without changing the other constants).

BenHenning · 2021-05-08T01:57:00Z

domain/src/main/java/org/oppia/android/domain/question/QuestionAssessmentCalculation.kt

+        allQuestionScores.size.toDouble() * maxScorePerQuestion / internalScoreMultiplyFactor
+    }.build()
+
+  private fun QuestionSessionMetrics.computeQuestionScore(): Int = if (!didViewSolution) {


Nit: suggest moving these below since they're separate from the private helpers of this class (they sort of belong to both this class & QuestionSessionMetrics as extension functions).

BenHenning · 2021-05-08T01:58:14Z

domain/src/main/java/org/oppia/android/domain/question/QuestionAssessmentProgressController.kt

@@ -56,6 +57,7 @@ class QuestionAssessmentProgressController @Inject constructor(

  private val progress = QuestionAssessmentProgress()
  private val progressLock = ReentrantLock()
+  @Inject internal lateinit var scoreCalculatorFactory: QuestionAssessmentCalculation.Factory


Does this need to be internal or can it be private? Prefer minimizing access to values when possible.

When I make the field private, Dagger complains and says "error: Dagger does not support injection into private fields", so that's why I made it internal. Do you know any workarounds for this?

BenHenning · 2021-05-08T01:58:59Z

domain/src/main/java/org/oppia/android/domain/question/QuestionAssessmentProgressController.kt

+      return dataProviders.createInMemoryDataProviderAsync(
+        "user_assessment_performance"
+      ) {
+        (this::retrieveUserAssessmentPerformanceAsync)(skillIdList)


Why can't this just be retrieveUserAssessmentPerformanceAsync(skillIdList)?

Fixed, thanks!

BenHenning · 2021-05-08T02:15:30Z

domain/src/main/java/org/oppia/android/domain/question/QuestionConstantsProvider.kt

@@ -11,6 +11,18 @@ annotation class QuestionCountPerTrainingSession
 @Qualifier
 annotation class QuestionTrainingSeed

+@Qualifier
+annotation class ViewHintScorePenalty


Each of these qualifiers should have documentation to explain what they correspond to/how to use them. E.g.:

/** Qualifier corresponding to the penalty users receive for each hint viewed in a practice session (an application-level injectable int). */ @Qualifier annotation class ViewHintScorePenalty

Added, thanks!

BenHenning · 2021-05-08T02:16:20Z

.../src/test/java/org/oppia/android/domain/question/QuestionAssessmentProgressControllerTest.kt

+    assertThat(hintAndSolution.correctAnswer.correctAnswer)
+      .isEqualTo("3.0")


Will these fit on the same line? If so, prefer that (in general, prefer to fill up as much line space as possible before line wrapping).

Ditto elsewhere.

BenHenning · 2021-05-08T02:21:43Z

.../src/test/java/org/oppia/android/domain/question/QuestionAssessmentProgressControllerTest.kt

@@ -994,6 +1003,515 @@ class QuestionAssessmentProgressControllerTest {
    assertThat(updatedState.ephemeralState.state.interaction.solution.solutionIsRevealed).isTrue()
  }

+  @Test
+  fun testRevealedSolution_forWrongAnswer_returnScore2OutOf3() {


Here & below: I suggest factoring high-level actions & their sanity checks into different helper methods to only keep the high-level details relevant to the behavior being tested in the method body. For example, with refactoring this test could instead be:

fun testRevealedSolution_forWrongAnswer_returnScore2OutOf3() { // Arrange setUpTestApplicationWithSeed(questionSeed = 0) subscribeToCurrentQuestionToAllowSessionToLoad() startTrainingSession(TEST_SKILL_ID_LIST_2) submitWrongAnswerForQuestion2() // NOTE TO JESSICA: this could also perform verification as a sanity check submitSolutionIsRevealed() submitCorrectAnswerForQuestion2() // Act val grade = getExpectedSuccessfulGrade() // Assert assertThat(grade.numerator).isEqualTo(2) assertThat(grade.denominator).isEqualTo(3) }

Notice how this can help focus the test body just to the details that the reader needs to see & hide the boilerplate to make it work. This is an easier test to verify, and reduces a lot of the overall code that we need to maintain for all of the tests being added.

I suggest making this or a similar clean up for all new tests here.

Refactored, thanks!

BenHenning · 2021-05-08T02:23:00Z

model/src/main/proto/question.proto

@@ -82,14 +82,20 @@ message AnsweredQuestionOutcome {
  bool is_correct_answer = 2;
 }

+// A user's grade for a practice session represented as a fraction.
+message FractionGrade {


Following up on this--can these values be ints now that everything is being scaled? You could always perform the final decimal calculation in the UI and keep things ints at the domain layer to simplify management.

Dismissing to unblock the PR since I'll be unavailable for reviews over the next 2 weeks.

rt4914 · 2021-05-10T18:55:06Z

@TheRealJessicaLi Assign me this PR once you have addressed Ben's and Vinita's comments so that I can cross verify them in my full PR pass. Thanks.

BenHenning · 2021-05-16T21:11:28Z

I won't be able to take a look at this until I return on or after the 24th of May but I'm happy to defer to @vinitamurthi for domain layer design decisions.

vinitamurthi · 2021-05-20T02:43:06Z

I will take a look once again to ensure Ben's comments are addressed, but otherwise the approach LGTM. @rt4914 can you please take a look? We are blocked on your codeowner review

vinitamurthi · 2021-05-22T12:15:52Z

LGTM from my side, @rt4914 PTAL

rt4914

LGTM for codeowners. Thanks @TheRealJessicaLi

TheRealJessicaLi added 2 commits April 3, 2021 21:48

added score calculations

6be0a7a

linted

2628b89

linted again more linting :) linted

TheRealJessicaLi requested review from BenHenning and vinitamurthi as code owners April 4, 2021 02:07

TheRealJessicaLi assigned TheRealJessicaLi, BenHenning and vinitamurthi Apr 4, 2021

vinitamurthi suggested changes Apr 6, 2021

View reviewed changes

vinitamurthi removed their assignment Apr 6, 2021

TheRealJessicaLi mentioned this pull request Apr 9, 2021

Implement Mastery Calculations for Practice Session #3067

Closed

addressed comments

d848ee0

TheRealJessicaLi requested a review from rt4914 as a code owner April 9, 2021 04:18

use BigDecimal to avoid floating point error

419ef0c

linted more linting

TheRealJessicaLi requested a review from vinitamurthi April 9, 2021 19:23

TheRealJessicaLi assigned vinitamurthi Apr 9, 2021

BenHenning previously requested changes Apr 13, 2021

View reviewed changes

BenHenning removed their assignment Apr 13, 2021

vinitamurthi suggested changes Apr 13, 2021

View reviewed changes

vinitamurthi removed their assignment Apr 13, 2021

TheRealJessicaLi added 4 commits April 24, 2021 20:04

fixed as per comments

c70a6e3

mock constants for other related tests

bf985f1

linted

7092c95

fixed merge conflicts

3816c7f

TheRealJessicaLi requested review from vinitamurthi and BenHenning April 25, 2021 01:36

TheRealJessicaLi assigned vinitamurthi Apr 25, 2021

BenHenning reviewed May 1, 2021

View reviewed changes

BenHenning removed their assignment May 1, 2021

TheRealJessicaLi added 2 commits May 1, 2021 15:15

made calculated scores immutable

06d3506

linted

e75cced

TheRealJessicaLi requested a review from BenHenning May 1, 2021 20:24

TheRealJessicaLi assigned BenHenning May 1, 2021

TheRealJessicaLi mentioned this pull request May 2, 2021

Fix #2738 and #3067: Mastery calculations and tests #3156

Merged

8 tasks

vinitamurthi approved these changes May 7, 2021

View reviewed changes

vinitamurthi removed their assignment May 7, 2021

BenHenning reviewed May 8, 2021

View reviewed changes

BenHenning assigned rt4914 and unassigned BenHenning May 8, 2021

rt4914 removed their assignment May 10, 2021

fixed

9ed68e2

TheRealJessicaLi requested review from BenHenning and vinitamurthi May 16, 2021 05:14

TheRealJessicaLi assigned rt4914, vinitamurthi and BenHenning May 16, 2021

rt4914 approved these changes May 22, 2021

View reviewed changes

rt4914 removed their assignment May 22, 2021

vinitamurthi approved these changes May 24, 2021

View reviewed changes

vinitamurthi merged commit 7adefe2 into oppia:develop May 24, 2021

	// TODO(#3067): create mastery calculations method
	// TODO(#3067): Create mastery calculations method.

	// TODO(#3067): set up finalMasteryPerSkillMapping
	// TODO(#3067): Set up finalMasteryPerSkillMapping

		assertThat(hintAndSolution.correctAnswer.correctAnswer)
		.isEqualTo("3.0")

Added score calculations #3040

Added score calculations #3040

Conversation

TheRealJessicaLi commented Apr 4, 2021

Explanation

Checklist

BenHenning commented Apr 6, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenHenning commented Apr 8, 2021

TheRealJessicaLi commented Apr 9, 2021

BenHenning commented Apr 10, 2021

BenHenning left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenHenning left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheRealJessicaLi commented May 1, 2021

BenHenning commented May 6, 2021

vinitamurthi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheRealJessicaLi May 15, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenHenning left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheRealJessicaLi May 15, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rt4914 commented May 10, 2021

BenHenning commented May 16, 2021 • edited Loading

vinitamurthi commented May 20, 2021

vinitamurthi commented May 22, 2021

rt4914 left a comment

Choose a reason for hiding this comment

TheRealJessicaLi May 15, 2021 •

edited

Loading

TheRealJessicaLi May 15, 2021 •

edited

Loading

BenHenning commented May 16, 2021 •

edited

Loading