RSVP-Style Gas Optimization Pots

[alex-ppg]

I would like to pose a high-impact suggestion in relation to the C4 process of contests as a whole, both as a warden and as a judge:

RSVP-Style Gas Optimization Pots

Brief

This proposal results in a change in the C4 contest structure with regard to the Gas Optimization pot; instead of having an open-for-all pot, the C4 org should transition to an RSVP-style Gas Optimization pot with a slight change in the process. There are multiple advantages that sponsors, wardens, and judges alike will benefit from.

As an addendum to this, I also propose a change in how the gas optimization reports are laid out to incentivize higher-quality submissions with tangible outputs for the sponsors. I will try my best to list both the pros and cons of these recommendations below. A concise TL;DR can be found at the bottom given that this particular issue is large in length.

Pros Per Role

Wardens

Given that the Gas Optimization pot is a distinct reward sum of a contest, a lot of wardens will feel the urge to submit a gas optimization report for a contest even if they are better suited to security analysis due to FOMO. This will result in a slightly more complex time-management schedule for wardens which they may not be able to uphold, leading to less time spent on securing a project.

Ultimately, as a warden juggles between allocating time to secure a project and allocating time to optimize a project one of the two submissions will be impacted, with gas optimizations being the impacted party more often than not.

Another issue with the current open-for-all pot is overlap. A significant portion of gas optimization reports contain overlap in the optimizations they recommend. This arises from the fact that while HM / QA submissions are debatable, gas optimizations are measurable and finite.

By being picked for an RSVP-style Gas Optimization pot, wardens will be clear in how they should manage their time and will also have a significantly higher reward, incentivizing them to greatly increase the quality of their reports.

Judges

One of the biggest bottlenecks of C4 contest as a whole is the time it takes to properly judge the vast majority of submissions a C4 project has. The crux of a C4 contest is the enhanced security it provides; Gas Optimization reports are a side feature that rewards wardens from a completely different pot.

A significant portion of wardens who would not submit HM / QA reports will participate in this dedicated pot, leading to an increase in the "default" judging time of a contest. Additionally, Gas Optimization reports tend to contain a lot of information, rendering judging them more time-consuming than reports of vulnerabilities due to the sheer amount of code segments that need to be evaluated in relation to the optimizations applied.

By transitioning to an RSVP-style Gas Optimization pot, the effort required to judge a contest will decrease leading to lower turn-around times for contests. Furthermore, judges would solely judge presumably high-quality submissions rather than needing to weed out bot-reported findings, automatic tool submissions, and other such spam/filler.

Sponsors

Sponsors would not actually need to acknowledge or judge optimization reports, they can simply consume them after the judging process concludes. As RSVP-style gas optimizations would solely contain tangible gas reduction recommendations, sponsors would be greatly inclined to consume each report and apply it to their codebase.

Finally, sponsors would rest easy knowing that their funds are being put to good use as they will always receive a valid, high-quality gas optimization report as an output of the C4 contest.

Judges + Wardens + Sponsors

Presently, gas optimizations are a pain point to judge due to the complexity that arises from them when they are ultimately applied. To elaborate, a gas optimization in paper does not necessarily translate in code due to the way the Solidity compiler already optimizes a lot of "inefficiencies" by itself.

This proposal (as explained further down in the process section) will result in:

• Wardens submitting gas optimization reports with "hard numbers" and measurable gas benefits directly linked to the compilation setup of the project
• Judges being able to evaluate submissions in a completely impartial way, evaluating the actual gas cost benefit that the sponsor acquires rather than a hypothetical one
• Sponsors would be confident in applying optimizations as they would unequivocally enhance their codebase

Cons Per Role

Wardens

As the subset of wardens eligible to participate in the gas optimization pot of a project would decrease, a portion of wardens would undoubtedly be impacted as they would no longer be able to participate in the pot. The greatest impact would be for grade B reports as grade A reporters would balance out or even increase their earnings by being picked for RSVPs.

Another potentially negative side-effect of these changes would be that RSVP wardens will shift their focus mostly to optimizing rather than securing a project. The more eyes on a codebase the merrier, however, I do not think this will have as big of an impact given that nothing will prevent RSVP wardens from submitting HM / QA reports and a warden excelling at optimizations does not necessarily mean that they excel at security and vice versa.

Judges

For judges and pre-sorts in particular, this change would result in a net positive. The effort required to judge / pre-sort a contest would decrease and while the judgment of an individual gas optimization report could increase, the number would be substantially lowered.

Sponsors

For sponsors, this change would result in a mostly positive impact. Their confidence in applying the optimizations of a warden would increase, and they would not need to waste time reviewing a bazillion number of reports for large projects that are marked as satisfactory.

The funds they have allocated for a contest would also be more optimally distributed and a sponsor would be confident that a guaranteed output of a C4 contest is a high-quality, applicable gas optimization report.

One potential drawback is the fact that certain optimizations may ultimately be missed by RSVP wardens. As the pot I am recommending changes for is solely related to gas-optimizations, I believe it is permissible for any optimizations to be missed as long as the RSVP wardens identify the largest and most impactful optimizations that can be applied to the codebase.

Process Changes

The process changes are ordered in the way they would be carried out in a contest to ensure that the text is cohesive.

C4-Wide

One main problem that exists for gas optimization reports (except from spam/bot/etc. submissions) are compiler-optimized statements vs. actual optimizations. The Solidity compiler is getting better and better at optimizing things, rendering certain "optimizations" having no impact if the optimizer is turned on, which it is for most projects in the space.

A very simple example of this is the optimization of keccak256 operations for the same payload. The compiler is smart enough to "optimize" consecutive lookups of the same mapping as a result. Example:

struct Bar {
    uint256 x;
}
mapping(uint256 => Bar) public foo;

function unoptimized() external {
    foo[0].x = foo[0].x + 1;
}

function optimized() external {
    Bar storage bar = foo[0];
    bar.x += 1;
}

The above example would result in GasTest::unoptimized having a higher gas cost than GasTest::optimized without optimizations whilst they would have the same gas costs with optimizations turned on for the compiler. (Tested w/ 0.8.7 & optimizations at 200 runs)

This renders it difficult to judge gas reports solely by their submissions, for example:

X	Gas Report A	Gas Report B
Total Recommendations	10	5
Applicable Recommendations	2	3
Best Report	?	?

In order to ensure gas optimization reports are judged in an impartial way, the ideal scenario would be to award the gas optimization report with the most compiler-specific applicable optimizations.

This is unrealistic in the current setup of C4 due to the vast amount of submissions as well as the significant increase in the judging time of a project this would presently take.

To this end, I propose a change in the gas optimization report process. Instead of permitting arbitrary optimization submissions, the RSVP wardens will need to provide rudimentary code examples (where applicable, like above) or a justification proving the optimizations they recommend will actually result in a gas cost decrease for the compiler setup of the sponsor.

As RSVP wardens are guaranteed to have a higher payout than in the previous model, increasing their effort with the above change in the submission process is acceptable in my eyes.

In the same way, judges would receive far fewer reports to actually evaluate as part of a C4 contest. Thus, increasing the effort required to judge a single submission is justifiable as the overall number of submissions would greatly decrease.

C4 Staff

In order to enforce this new RSVP-style gas optimization contest adjustment, the C4 entity will need to maintain a list of wardens that have proven themselves capable of optimizing a codebase significantly. The criteria for this are debatable, however, C4 does have all the data necessary to make a preliminary list of these wardens. I propose that this role is set as optimizoor.

Once this list has been composed, each upcoming contest will have a period whereby optimizoor members can apply to become the optimizers of the particular contest. The number of optimizers that should be permitted per contest would be a minimum of 1 dedicated optimizoor and up to 3, however, this is, of course, up for discussion.

The reasoning behind a minimum of 1 is because the list of optimizoor members will need to increase somehow organically. Creating an application process is not really optimal effort-wise for the C4 entity, so what I propose is a single (or multiple) "invitational" slot(s) per contest.

Basically, each new contest will have a set of dedicated optimizoor wardens as well as some "new" wardens who wish to become optimizoor members. These wardens would apply to become an optimizoor by actually participating in the gas optimization pot of a contest.

If their submission is deemed as grade A, they would acquire the optimizoor role and be eligible to RSVP for future contests.

If their submission is not deemed grade A, they would be barred from re-applying for a set period of time to disincentivize spam and also open up the invitational slots for other wardens who may be more capable.

Picking the wardens that should participate in the "invitational" should be relatively straightforward; wardens that have historically participated in optimization pots would be preferred over fresh wardens. I believe we can further ideate on how to make this process fair (for example, having one invitational slot randomly assigned whilst the other would follow the past-contest rule), however, I think an organic way for the optimizoor role set to increase similarly to certified wardens is warranted.

Sponsors

During the contest request phase, a sponsor will need to provide precise compilation instructions for their project. While this is presently done, it should be added as a dedicated chapter to the README.md file of a contest.

Wardens

Due to the RSVP-style manner of the contest, wardens will have to put more effort into their gas report submissions than under the previous contest style.

Each gas optimization they recommend will need to be accompanied by either a brief "PoC" that illustrates the optimization in the compiler settings of the project or by a chapter that details why the optimization is applicable.

Judges

Judges would be instructed to fully vet the submission of each warden and properly test the code examples (if any) a warden provides in a test environment such as Remix.

As is the case with HM / QA reports, a judge will have the final say in which gas optimization report they find to be best. However, judges are urged to pick the optimization report with the highest impact for the sponsor when applicable.

TL;DR

• Wardens submitting gas optimization reports with "hard numbers" and measurable gas benefits directly linked to the compilation setup of the project
• Increase of competition for the gas optimization pot with higher rewards for RSVP wardens
• Judges being able to evaluate submissions in a completely impartial way, evaluating the actual (to the extent possible) gas cost benefit that the sponsor acquires rather than a hypothetical one
• Judges reducing the time it takes to judge a contest by "eliminating" low and medium-quality gas-related submissions
• Sponsors being confident in applying optimizations as they would unequivocally enhance their codebase
• Sponsors being guaranteed a high-quality, applicable gas optimization report as a result of a C4 contest

Closing Thoughts

I sincerely believe that the above change would increase the competition for Gas Optimization reports in a healthy way, ensure that optimization-centric wardens are aptly awarded in a C4 contest, and improve the expectations of a C4 contest by ensuring that a high-quality gas optimization report is an actual and not potential output of a C4 contest.

[IllIllI000]

What percentage of the total possible gas-saving optimizations, would you say, are currently reported by the top bots in the bot races? It seems that lately the gas reports excluding those findings have been much smaller.

[alex-ppg]

Thanks for contributing @IllIllI000! A significant portion of gas-saving optimizations are indeed identified by the bots, however, these optimizations tend to either be false positives, low-impact findings, and/or significantly impactful to the code-style of the code.

I invite other, more seasoned judges to provide a more historically accurate response to your question; in my most recent personal experience, the bot report provided only one valid and significant recommendation that the sponsor should have applied whilst wardens that competed identified multiple optimizations that were not detected by the bots.

I think there is merit in f.e. actually permitting these RSVP-style gas optimization reports to reference bot findings that are applicable for the sponsor rather than penalizing these reports. I am quite confident that sponsors will not go through all results of the bot report if they start seeing false positives, however, they would be much more inclined to apply them if a human has post-processed the outputs, kind of like how auditors filter static-analysis outputs manually.

Another aspect to consider is the actual benefit they result in. Bot reports tend to list a lot of sub-50 gas per operation optimizations that are also sometimes optimized by the compiler automatically. Manual warden optimization reports (grade A) tend to contain high-impact and curated recommendations rather than generic advice that may ultimately be inapplicable.

[GalloDaSballo]

I'm unable to give a longer answer because I believe the suggestion has a lot of different points.

I think overall C4 should be willing to offer the service, but it would have to be driven by customers requesting it.

And since C4 can always get a few Wardens to "freelance" it shouldn't need to define a specific service unless there's demand for it.

Ultimately if a Sponsor wants a Gas Audit, C4 will for sure offer them and will create rules to allow that to be done at the highest level.

In terms of evaluating the work, it will have to be a mix of:

• Real impact (from Sponsor / Org Provided Tests) - Quantitative Analysis
• Maintainability
• Safety / Effort

The other 2 aspects are more qualitative and would be hard to quantify