Proposal: Prevent overuse of "burden of proof" to increase objectivity of judging

[xuwinnie]

Wardens have the burden of proof in submissions. Explaining and rationalizing the potential impact is an essential part of a quality submission. The burden of proof increases based on the potential value of the submission (rarity, severity).

The burden of proof is established to set a minimum threshold for the quality of reports. However, in some cases, judges use it to invalidate issues without providing any explanation. To ensure our long-term success, especially with the increasing number of appointed judges, we cannot solely rely on the judges' integrity. A system of check and balance could be helpful as we may decentralize our judging process in the future.

Therefore, I propose that if an issue meets two criteria: 1. Deemed of sufficient quality by the lookout. 2. Confirmed by the sponsor. If a judge decides to invalidate such an issue, the burden of proof should shift to the judge, who must explain their decision. Naturally, we need to ensure that judge and lookout cannot be the same person.

In fact, in 99% of cases, the content in proposals is already being implemented. However, formalizing these practices into the rules can enhance the objectivity and fairness of the outcomes at a higher level.

[MiloTruck]

The problem with this rule is when submissions have duplicates, since:

• The low/sufficient/high quality labels do not apply to individual submissions. I don't think I've ever seen an issue marked as "low quality" duped to a "sufficient quality" one.
• The sponsor only checks primary issues.

This makes it possible for an extremely poorly written report to be duped to a higher quality primary issue. In this scenario, the criteria above doesn't really make sense, since the "sufficient quality" and "sponsor confirmed" labels were for the primary issue, not the poorly written duplicate.

Also, if a lookout is unsure whether a finding is valid or not, even if it is poorly written, marking it as "sufficient quality" is favored due to the asymmetric risk of missing a valid finding vs making the sponsor review more spam. Therefore, such labels might not be an accurate reflection of a report's quality.

[xuwinnie]

@MiloTruck I understand your concern. Let's discuss two cases.

1. The report does not has duplicate. Then it is a solo which sponsor has confirmed. The judge can still invalidate it, but I think it makes sense to ask the judge to provide reasons here.
2. The report has duplicate, and we can encourage lookout to mark it as low quality if it is low quality. (let the label apply to each submission).

The judge still has final authority on the result, the proposal just aims to prevent abusing "burden of proof" to invalidate confirmed submissions with no reason.

[GalloDaSballo]

Please share some examples of this (here or privately if from backstage)

[xuwinnie]

Please share some examples of this (here or privately if from backstage)

code-423n4/2023-07-tapioca-findings#1379 This is an example.

[bytes032]

The Lybra finance contest is a prime example. The judge invalidated numerous issues for lacking (PoC), citing the burden of proof.

Meanwhile, the project has zero tests and you've got to set up your own test fixture just to get it working.