Validator workflows: what's next #162
CloudEllie
announced in
Announcements
Replies: 1 comment
-
Footnote re: Validator accuracy The original concept for Validators proposed:
However, in practice, we learned a few important things:
|
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
After weighing the benefits and challenges of the current Validator process, we've decided to make some changes to the Validator role:
This change will apply to audits starting on or after Monday, July 8.
Why change?
The initial Validator concept was quickly proven unsatisfactory as it did not handle deduping effectively. As such, we pivoted to having all 3 Validators review all submissions -- which greatly increased the effort required, for the same split award pool. The compensation-to-effort ratio has been a challenge, especially given the speed at which Validators are expected to work. As a result, RSVPs have been understandably low, and staff have had difficulty finding people to perform the work.
It's also not clear that the results produced by having 3 validators justify the additional effort involved.
Key learnings
Predicted challenges in shifting to a single validator:
The first challenge is addressed in the "Planned adjustments" section below. The second is more difficult to mitigate as QA judging guidelines remain emergent. I expect that we will just need to observe and learn as we go.
What stays the same:
Planned adjustments
We intend to make the following adjustments to the (solo) validator role:
a. reviewing the judge's decisions in the findings repo, and
b. re-reviewing submissions that were marked unsatisfactory during the validation phase (i.e. not moved to the findings repo) and that were submitted by non-SR wardens, to ensure that any duplicates of valid issues are "rescued." The intent here is to reduce false negatives, and to ensure that SR wardens do not abuse their role by arguing for their own submissions, but ignoring other warden's duplicates.
Beta Was this translation helpful? Give feedback.
All reactions