Supreme Court verdicts, Autumn 2023: post-judging QA

[CloudEllie]

Code4rena's inaugural Supreme Court session (Autumn 2023) has reached the "post-judging QA" phase.

A draft of all Supreme Court verdicts for this session can be viewed here:
https://docs.google.com/document/d/1Y2wJVt0d2URv8Pptmo7JqNd0DuPk_qF9EPJAj3iSQiE/

Community members are welcome to post comments and questions here. To keep discussion structured, please:

• Check to see if a comment thread exists for the verdict you're discussing; if so, please thread your comments within that thread.
• Start new comment threads with the title and page number of the verdict you wish to discuss (e.g. "Using contract upgradability as a severity mitigation - p. 10")
• The usual post-judging QA rules apply here, namely: All comments should present fact-based evidence, rather than opinion.
• Do not re-state facts or arguments that are already present in the org issues; those have all been reviewed in depth as part of the court's deliberation process.

Huge thanks to our Supreme Court judges: @GalloDaSballo (Alex the Entreprenerd), @dmvt (LSDan), and @trust1995 (Trust), who have done an extraordinary job of reviewing C4 "case law" and proposing improvements to rules, judging guidelines, and processes. 👏 👏 👏

Edited to add: Post-judging QA will close Friday, November 17, 2023 at 18:00 UTC.
Further edited to note post-judging QA rules.

[JeffCX]

Thanks the Supreme Court judges for drafting such a comprehensive verdict

Want to seek more clarity on a few points

I wonder which section addresses the proposal to seek clarity on whether POC or additional info and proof submitted during post judging QA can used as additional supplyment for orignal report to escalate severity

Original thread

#122

I am not sure how the statement rules whether additional proof submitted should be considered by judge

[JeffCX]

After reading the doc again, I find this

would love to seek more clarity on that

Elaborations of the already introduced information can be considered (e.g. tweaking a POC)

I think that is open to intrepret and difficult to judge what is the border between Elaborations and introducing new info

but they will only count towards the validity of the issue, not its quality score.

emm what does the quality score mean here? is this mean if user Elaboration on POC, judge should not remove 50% duplicate and give full credit?

but if the Elaboration with valid point, issue can be changed from invalid to valid or medium to high??

[dmvt]

Our point in issuing these rulings is not to provide a concrete decision on all future issues, but a framework inside of which the contest's judge can evaluate submissions. As before, if you want to ensure you get a high score as a warden, the best way is to provide a high quality submission.

emm what does the quality score mean here? is this mean if user Elaboration on POC, judge should not remove 50% duplicate and give full credit?

What we are stating here is that an invalid issue may become valid with the clarification of a POC, however, the submission's ultimate quality will be judged based on the original submission's quality, not taking into account additional clarification provided. If an issue is of low quality to begin with, a clarification of POC will not change this. If it was ruled invalid because the POC was unclear, a clarification of the POC may cause it to become valid.

but if the Elaboration with valid point, issue can be changed from invalid to valid or medium to high??

invalid -> valid

[JeffCX]

That make sense a lot! Thanks!

[GalloDaSballo]

A POC added in PJQA could only offer more context, but not more information as otherwise that would be abusable.

[JeffCX]

Emm wonder if anyone can provide an example where in c4, a OOS library / contract impact in-scope contract but the issue should not be rewarded

is that mean for example

if protocol use a vulnerable openzeppelin library that allows a attacker to initialize the implementation and self-destruct the contract to lock fund, the issue is invalid?

also,

if a contract consume a oracle that is vulnerable to read-only entrancy, we are argue the reentrancy happens out side of the in-scope contract, or we can argue the protocol should validate the read-only reentrancy before consume the price, so is the report valid or invalid?

based on this rule it is OOS library impact in-scope contract as well

so would love to clarity more clarity

[dmvt]

In all scenarios where there is a lack of clarity, the contest's judge will have the final say. Assuming it is implemented, the warden will have the opportunity to escalate the issue to the Appeal Committee after PJQA. If further clarity is required, it will be ruled on during the next SC session. If this is something that is near and dear to you, I recommend you collect data between now and then to present to the court.

[CloudEllie]

From a staff perspective, it feels important to differentiate between two purposes of scope definition:

1. Defining the scope of work for the audit, so that both sponsor and wardens have clear agreement and understanding of the work that's expected to be performed
2. Defining the full scope of all potential threats to a codebase, for the purpose of securing a project.

For many reasons, Code4rena's audits are scoped with a focus on (1), although of course best practices in security mean that (2) should always be considered.

But for Code4rena's processes, judging, and awarding to function fairly, the only practical approach is to define clearly what code is in-scope, and what is out of scope for the purposes of each audit.

Many C4 sponsors have bug bounties or informal processes for rewarding out-of-scope bugs, and C4 staff do our best to direct wardens to such avenues where applicable, if they discover a valid and valuable bug that falls outside the scope of a C4 audit. IMO this is the preferred approach to ensure that everyone's time and energy during the audit is well spent.

[GalloDaSballo]

I would like the Devs + Sales team to also read the above + the SC Verdicts Document as this clarification implies that the Sponsor must be made aware of how Scope is determined

[JeffCX]

delegate reports are out so I can refer to one of my past finding

under the rule above,

is this mean if sponsor does not define opensea contract in-scope,

code-423n4/2023-09-delegate-findings#94

this find is not valid?

so it is sponsor's job to define the scope of how external contract impact in-scope code?

[GalloDaSballo]

The finding seems to be in scope:
https://github.com/code-423n4/2023-09-delegate/blob/main/scope.txt

[JeffCX]

these are great metric, wonder what is the purpose of collecting such data

I mean c4 stuff can have access to this data to know who judges / presort better, so will this metric to be used to determine who will be selected for judging / presort in case multiple judge / lookout interest in help out one contest?

besides that, if this data disclosed paired with who judges the contest prior to the contest starts, that is useful for warden as well

these are great metric but just wonder how to consume it to make it more useful,

c4 can consume it to select judge that judge betters

but disclose such data can be useful for warden as well

[JeffCX]

this is related as well

wonder what is the reason to not recommend

I still propose we should let warden vote on that proposal

original thread:

#124

[CloudEllie]

re: deciding by vote - I would recommend reading "How consensus is built" in the Code4rena docs.

[dmvt]

these are great metric, wonder what is the purpose of collecting such data

I mean c4 stuff can have access to this data to know who judges / presort better, so will this metric to be used to determine who will be selected for judging / presort in case multiple judge / lookout interest in help out one contest?

We intentionally avoided making inferences about what those data will mean, preferring instead to first look at patterns in the initially collected set.

wonder what is the reason to not recommend

It is our view that disclosing this information will skew the results of contests

[trust1995]

Hoping our work in standardizing the rulebook will assist judges, sponsors and wardens in performing their duties and create healthy incentives for all.

Regarding Verdicts QA, would like to separate between our ruling output and consultative output. Anything that isn't a rule change is not viewed in scope for QA requests, it is targeted for the staff to consider how to proceed. Feedback is of course welcome as always.

For rulings, before commenting we would like to request that wardens read the entire document, as some topics are answered under other titles. We've covered all non-spam gov requests.

Best of luck to all.

[0xJuancito]

Thanks for the report and the efforts!

I have two questions:

1. How will missing duplicates "escalation" be treated?

• I believe it's the most common type of "error" by the judges/lookouts (any stats would be appreciated).
  • On the judges side, that may imply lots of fees payments on misses
  • On the warden side, that would incentivize to send more "gray" findings as Mediums instead of Lows, as they are generally more noticeable
• Also, the amount of time to check if an issue is a duplicate of another is considerably lower than to check the validity/impact of an independent issue. Maybe it would make sense to treat them in a different way.

2. Could you provide clarification on C4 collects a 10 escalation deposit, $3k?

Doc says about wardens:

Escalation costs $500 up front

It looks like an inconsistency, or I am missing something?

[dmvt]

I politely suggest that you attempt to collect this data and add it to an issue which can be reviewed in the next SC session.

The $3k fee is to cover the cost of the Appeals committee in the case of a 2/3rds consensus (the middle case):

[JeffCX]

Love this appeal committee mechanism

I suggest we can add two more requirement

• the initial judge should not be in appeal commitee
• it is highly recommend for warden to provide a coded POC when escalate, this save judge time and warden's money

[trust1995]

1. This was the intention of the court. We've now explicitly added it to the document
2. We are against the motion. The appeal decides if the judge made the right call, not if the submission is valid. Therefore, only information available during PJQA will be taken into consideration. We acknowledge this wasn't obvious in the draft language and updated it now.

[CloudEllie]

I've edited the original post to clarify the rules of post-judging QA. I'd also underline this comment from @trust1995:

Regarding Verdicts QA, would like to separate between our ruling output and consultative output. Anything that isn't a rule change is not viewed in scope for QA requests, it is targeted for the staff to consider how to proceed.

[osmanozdemir1]

Hi,
Thanks for amazing efforts and the report.

I would like to share my thoughts about "Standardize acceptance of reports based on automated findings - page 9".

I have to say I am uncertain about how protocols handle bot findings, but I don't believe they address all low-severity issues identified by the bots. Frankly, I think the majority of these low-risk bot findings remain unfixed by the protocols.

The current verdict regarding issues based on bot findings assumes that these bot findings will definitely be fixed and invalidates H/M issues with the reasoning that the "fix is the same". However, this verdict may result in unsubmitted and unfixed high severity bugs, as wardens will not delve deep into those low-severity bot issues due to the absence of incentives.

I believe if a warden shows High or Medium impact for a low severity bot finding, it should be considered valid even if the fix is the same. Otherwise, no-one will search and submit those H/M issues, which will create false sense of security, and the sponsor will most likely not fix the underlying low severity bot finding.

As I said above, I don't know exactly how protocols approach to these bot findings. You can forget all the things I said if protocols usually fix all of these bot findings. Otherwise, showing a higher impact (regardless of the fix being the same or not) might be a decision-changing added value to the sponsor.

I would like to hear your opinions on this matter.
Thanks

[IllIllI000]

principal of least sponsor surprise would dictate that all findings in an audit report need to be reported as the most severe severity for each applicable case, and excluding findings proven to be of higher severity than a bot race finding, in favor of spam finding reduction, sounds surprising

[GalloDaSballo]

Need more thoughts

[lokithe5th]

Dear judges, thank you for your hard work on these matters.

When the only prerequisite for a high or medium issue is a Bot Race finding, and fixing the relevant bot race finding in the most logical common way will cause the issue to no longer exist, the issue is invalid.

In my personal opinion, if a warden is able to show how a low- or medium-severity finding, which has been identified during a bot race, can be used in conjunction with some other nuance of the protocol's implementation (not necessarily combined with another bug) to produce a higher than already reported issue, then the finding has validity, even if fixing the lower-severity issue would have closed off the attack vector.

My reasoning:

The reported severity of an issue has a direct impact on the sponsor's motivation to fix that issue. Fixing low impact issues may not be a priority with the sponsor's resources.

Sponsors pay for the bot race and can be expected to take results seriously.

We (devs) use these labels as a way to filter the most pressing issues. All projects accrue technical debt. A sponsor needing to allocate resources is likely to look at a "low" severity rounding issue (for example) and dismiss it as a "nice-to-fix". The entire reporting system is aimed at prioritizing fixes.

By dissuading wardens from developing possible attack vectors to their fullest, the sponsor may remain vulnerable to issues in the code because they are deemed "low-risk", and thus "low-priority".

A warden taking the risk to successfully develop a higher-severity attack than what is reported, will be highlighting an issue that bad actors will take advantage of. This makes it a higher priority to fix and is, ultimately, valuable to the sponsor and the industry.

[GalloDaSballo]

If we only used 4nalyz3r to determine known issues then the above scenario would most often never happen.

The few times it would, it would spark a conversation around the specific selector which would most likely be removed from the Bot as a way to avoid the gotcha.

The conversation can be extended to how a Known Issue from Sponsors should be treated where it does make sense to sometimes tell the Sponsor "You're wrong" as a way to escalate potential risks

Interested in more thoughts by @dmvt @trust1995 @lokithe5th @sockdrawermoney

[lokithe5th]

@GalloDaSballo touches on something important here:
In the business relationship, C4 is the expert providing security services to the sponsor.

With deep respect, I am wary of the statements "rational fix must be assumed" and "anything reported in the bot race is assumed to be fixed correctly". This assumes the sponsor will treat the (likely) low-severity bot findings with the same urgency they will give a high- or medium-severity finding.

And yet, from C4's docs:

QA (Quality Assurance) Includes both Non-critical (code style, clarity, syntax, versioning, off-chain monitoring (events, etc) and Low risk (e.g. assets are not at risk: state handling, function incorrect as to spec, issues with comments). Excludes Gas optimizations, which are submitted and judged separately.

Given this classification of severity, in my humble opinion, we cannot responsibly assume that a sponsor will fix all the issues we have marked as QA. This is the practical reality that most protocols wrestle with: "must-fix", "nice-to-fix", "won't-fix"

To offset this tendency to prioritize according to severity and protect against false-negatives (where we accidentally under-report the severity of an issue), wardens should be able to develop issues to find the highest severity issue that they can. They do this at their own risk, mindful of their own EV. This draws upon the potential of bots as powerful screening tools to assist wardens in finding critical vulnerabilities; instead of an obstacle to a thorough security review.

This is also in line with the sentiment expressed in the SC verdicts on duplicate issues:

The findings are duplicates if they share the same root cause.
More specifically, if fixing the Root Cause (in a reasonable manner) would cause the finding to no longer be exploitable, then the findings are duplicates.

Given the above, when similar exploits would demonstrate different impacts, the highest, most irreversible would be the one used for scoring the finding. Duplicates of the finding will be graded based on the achieved impact relative to the Submission Chosen for Report.

Allowing bot race findings to be used as the basis for higher-severity findings seems to offer the greatest upside.

Where this relates to Known Issues, again in keeping with the role the wardens and C4 play in this relationship, I would argue that submissions where it can be demonstrated that a known issue has an impact/effect that was not acknowledged and due to this is of higher-than-acknowledged severity, that such an issue should be valid. The value to the sponsor lies in the security expertise they receive through C4 and the wardens. It is the wardens' role to test the sponsor's assumptions about such known issues.

This might mean that we prioritize the security of the sponsor at the cost of more complicated judging (acknowledging here the extremely difficult task that judges have).

The alternative would be to allow a sponsor to deploy with an issue which they think is non-critical but in reality leaves them exposed to risk. This breaks a key invariant: the security of the sponsor's code is paramount.

[aktech297]

Urgent!!! Result can be announced anytime..

Thanks for creating such a the wonderful platform. Much needed to iron out some mistakes.

I would to get judges feedback decision on this issue. I have put some comments.

https://github.com/code-423n4/2023-08-chainlink-findings/issues/832

[trust1995]

Hi,
The QA round is for feedback around the SC decisions. Any discussion around changing verdicts on running contests is out of scope.

[IllIllI000]

More specifically, if fixing the Root Cause (in a reasonable manner) would cause the finding to no longer be exploitable, then the findings are duplicates.

Does that mean the separate issues of 'transfer() return code not checked' and 'USDT signature mismatch' are duplicates, because they can both be fixed by using safetransfer() (which internally has multiple fixes)?

[dmvt]

Not duplicates per se, but the fix of one invalidates the other. More importantly, these should both be caught by the bot race making this distinction irrelevant. Even if they aren't, I suspect USDT will be excluded by almost every sponsor going forward due the the nonstandard erc20 recommendations.

[GalloDaSballo]

I would argue that they are for Contests, assuming a rational mitigation, but can be sent separately for Bot Races

[IllIllI000]

The exception is an actual threat to the functionality of the protocol or funds thereof.

Is this meaning to say that if you show a code path where funds are lost, that it can be med/high (i.e. that you're just trying to exclude findings that just say 'fee on transfer problem here!' and nothing else)?

[dmvt]

That's up to the contest's judge. It would have to meet the "actual threat" threshold, which "FoT doesn't work" on its own would not.

[IllIllI000]

Interpreting this rule should be done in favor of the Sponsor as to avoid breaking the intention of this ruling.

I don't understand what specifically this is trying to say - can you rephrase? In favor of the sponsor paying less? What is the intention that could be broken?

[trust1995]

The intention is to avoid breaking the one award / listed finding per bug invariant.
Sponsor has already paid for the contest, that's not a factor.
Allowing it opens the door for perverse incentives and scam submissions.

[alcueca]

There is a thread up there that mentions the appeal committee mechanism, but I'll start a new one for cleanliness.

From the fundamentals, I think that the sponsor shouldn't fund the court of appeals, much less an unknown value (even within a range). It just doesn't seem fair, or good sponsor experience. Even being a small amount, it seems an unnecessary point of friction.

There are a few possible ways of removing the sponsor from the mechanism. The one I like best is that in the case of a 2/3 ruling against the judge the court fee is paid 50/50 by the warden and the judge.

[alcueca]

If it is not a 3/3 ruling, it is not that clear that the warden is completely free of blame. I think that splitting the court costs between the warden and the judge is fair, since the judge wasn't that far in the wrong either.

Note that I propose this even though it will decrease my own rewards, in comparison with the system originally proposed. That's how much I think the original system is wrong with regards to the sponsors.

[lokithe5th]

I would second @alcueca's proposal, although it appears harsh at face value.

My reasoning would be:

1. Expecting the sponsor to front a deposit for possible escalations, when the sponsor has no say in the resolution of those escalations (no seat at the table), seems misaligned.
2. From the suggestion above, in the case of a 2/3 verdict in the warden's favor the court fee of $300 is paid 50/50 by the warden and judge. This would mean the warden receives $350 back from their initial $500 deposit. The total cost of the appeal for the warden is then $150 in such a case. Although this (and the $500 escalation cost) seems a high barrier, especially to newer wardens, it creates a good incentive. If the net EV of a different verdict for a warden is not >$150, then it is unlikely to be worthwhile for the judges to deliberate on it again.

Just as a general note, as it is not expressly prohibited, I foresee that the escalation cost/risk will probably be carried by multiple affected wardens through nominating another warden, in cases raised through this appeal mechanism. Such a group mechanism effectively allows most wardens to coordinate and participate in cases where the net EV is <$150 individually.

[0xA5DF]

A third option might be to create a pot to fund those cases.
The pot would come from an additional fee that the sponsor would pay for an audit and can be adjusted depending on the balance of the pot (i. e. if the pot reached 10K then we might decrease this fee).

This would incur an additional cost to the sponsor but I think it'd be better than requiring the sponsor to take the risk of having a much larger cost.

[GalloDaSballo]

The main downside of this is that now EV would be the biggest factor to determine ones escalations.

We can expect scenarios in which a Warden would escalate because it's +EV and others in which they wouldn't because in-spite of being right they would lose value in most circumnstances

[alcueca]

Think about if from the sponsor point of view. Imagine that in a contest a bunch of issues get escalated to the SC, and decided on 2/3.

1. Whether they are right or not, sponsors like contest results with lower severities. Due to SC appeals, their report has now higher severity vulnerabilities.
2. A bunch of judge decisions got appealed and overturned. Now the sponsor thinks they got a not so good judge, and therefore a not so good contest.
3. And because they got a bad judge, the sponsor has to pay extra money to get a report that paints them in a worse light.

I'm not saying that we should start rubber-stamping protocols to make sponsors happy, but at least we could avoid making them pay for our mistakes, more so when those mistakes are already unpleasant to them.