How are Gas/QA reports ranked?

[IllIllI000]

Now that a few awards with the new methodology have been released, it would be good to hear how the judges approached the ranking of the consolidated reports. The current guidance says that they're graded on a scale but aside from that there are very few details. Here are a couple of specific questions I had:

• Are judges ranking gas reports based on the approximate total gas saved, or on the number of classes gas findings? Same sort of question for QA reports (number of low/borderline medium issues, or total findings).
• If a finding can fall into both reports if worded differently, is one excluded or are they counted separately?
• Have there been any commonalities in what caused the top three reports in each category to place the way they did?
• It seems like issues that were downgraded from medium tend to bump up the QA report's ranking - is that the case? How do you decide how much more valuable a downgraded finding is than the sum of the other findings submitted separately?

For the excluded/separately question I would think that sponsors would want to hear both reasons for making the change, so I think having the differently-worded issue in both would be best. What do others think? What has been the approach taken so far for these questions? Any other questions?

[IllIllI000]

Alex The Entreprenerd

I am rating Gas findings exactly by gas saved (estimate subject to human error) and then adding points or removing based on the quality of the writing

[IllIllI000]

harleythedogC4

Now, here is the methodology I used for calculating a score for each QA report. I first assigned each submission to be either non-critical (1 point), very-low-critical (5 points) or low-critical (10 points), depending on how severe/useful the issue is. The score of a QA report is the sum of these points, divided by the maximum number of points achieved by a QA report. This maximum number was 26 points, achieved by code-423n4/2022-02-nested-findings#66.

The number of points achieved by this report is 26 points.
Thus the final score of this QA report is (26/26)*100 = 100.

[IllIllI000]

No answers about the second bullet point in my original post as of yet

[IllIllI000]

Alex The Entreprenerd

After adding code-423n4/2022-02-tribe-turbo-findings#42 , because of the exceptional detail and presentation, I believe 6/10 to be the proper score.

Ultimately had the Warden added a few of the lower importance finding I believe this would have easily been the best report from the contest

So judges are following different methodologies. As long as the method is documented, seems fine. There are other reports that don't have any explanations though

[IllIllI000]

harleythedogC4

Now, here is the methodology I used for calculating a score for each gas report. I first assigned each submission to be either small-optimization (1 point), medium-optimization (5 points) or large-optimization (10 points), depending on how useful the optimization is. The score of a gas report is the sum of these points, divided by the maximum number of points achieved by a gas report. This maximum number was 10 points, achieved by code-423n4/2022-02-nested-findings#67.

The number of points achieved by this report is 10 points.
Thus the final score of this gas report is (10/10)*100 = 100.

[IllIllI000]

Alex the Entreprenerd

Best submission 8/10
To improve:

• Actual Gas Savings math (sort findings by impact)
• List of all spots to fix (So the sponsor can implement instead of it being a puzzle)

Alex the Entreprenerd

I think all submission should try to show the code and have titles, it does help.
::paragraph of gas savings explanation::
225 gas saved

Alex the Entreprenerd

The report is great, gives some general considerations as well as specific advice to implement.
Great submission.
Only negative note is the warden missed re-entrancy, beside that, this is how I think a QA report should be done

Alberto

Unadjusted score: 30 (low risk of something going wrong x3, which is the maximum multiplier for instances found).

Alberto

Unadjusted score: 55 (inclidung 30 points for the extensive architecture recommendations).
Extra 5 points from # 24

Alberto

Unadjusted score: 50 (Including 20 points for nice formatting)

Alberto

The slot is warm, but still you should save about 300 gas.
Score: 300

Alberto

Great report. Score: 4300 (including +20% from formatting)

gzeon

This report covered most of the gas optimization but is quite difficult to consume due to its length and missing an index. Probably don't need to include all the code snippet in the report.

moose-code

Really like the S M N report nomenclature. Very true that gas optimisations are not one size fits all. I mean, we could write the whole thing in opcodes but readability is destroyed.

[IllIllI000]

accidentally closed it

[Rassska]

Huge kudos for that :D

[IllIllI000]

gititGoro

For the QA, I have a spreadsheet with a series of criteria such as documentation quality, importance of finding and so on. Each category has a 1-5 rating and the numbers have clear meaning. For instance, 1 - poor effort but explains the problem, 2 - explains the problem and links to code, 3 - links and provides POC etc.

Then I have a formula that weights the columns according to importance and produces a sum. The highest scoring warden is then weighted as a 100 and everyone else as a percentage of that and you get your curve.

[IllIllI000]

jack-the-pug

top-notch formatting. I love how you put a short and clear description for each issue, with all the instances listed.