RFC: Bot submission standardization #118
Replies: 19 comments 37 replies
-
The category example should be |
Beta Was this translation helpful? Give feedback.
-
Great proposal, I like it. It's not entirely clear to me what is the expected value of "content" for each instance. Could that be a code snippet, comments, any text content, or is it something more specific? |
Beta Was this translation helpful? Give feedback.
-
Can add below items to the standard:
Judging side: |
Beta Was this translation helpful? Give feedback.
-
I would also like some sort of free-form data field for the report as a whole, for things like my rubric section |
Beta Was this translation helpful? Give feedback.
-
Great idea, 2 notes:
|
Beta Was this translation helpful? Give feedback.
-
Good idea! I proposed also that "scope file list" should have a standard form, and we could take this change in order to standardize input and output. |
Beta Was this translation helpful? Give feedback.
-
Great idea, |
Beta Was this translation helpful? Give feedback.
-
We absolutely need one reference for scope of contest and do not change it when is running. |
Beta Was this translation helpful? Give feedback.
-
Some comments/questions:
|
Beta Was this translation helpful? Give feedback.
-
My recommendation: {
"summary": [
{
"severity": "H",
"issues": [
{
"title": "Issue 1",
"instances": "10"
},
{
"title": "Issue 2",
"instances": "20"
}
],
"extra": "Total: 30 instances over 2 issues"
},
{
"severity": "M",
"description": "{{description}}",
"issues": [
{
"title": "{{issue title}}",
"instances": "{{number of instances}}"
},
{
"title": "{{issue title}}",
"instances": "{{number of instances}}"
}
],
"extra": "{{extra information}}"
},
{
"severity": "G",
"description": "Gas totals use lower bounds of ranges and count two iterations of each for-loop. All values above are runtime, not deployment, values; deployment values are listed in the individual issue descriptions. The table above as well as its gas numbers do not include any of the excluded findings.",
"issues": [
{
"title": "Issue 1",
"instances": "10",
"saved": "100"
},
{
"title": "Issue 2",
"instances": "20",
"saved": "200"
},
{
"title": "Issue 3",
"instances": "20",
"saved": "-"
}
],
"extra": "Total: 50 instances over 3 issues with **300 gas** saved"
}
],
"findings": [
{
"severity": "H",
"issues": [
{
"title": "Issue 1",
"description": "Description of issue 1",
"instances": [
{
"link": "https://github.com/code-423n4/2023-10-zksync/blob/main/code/contracts/ethereum/contracts/zksync/facets/Mailbox.sol#L37-L37",
"content": "```solidity\nFile: code/contracts/ethereum/contracts/zksync/facets/Mailbox.sol\n\n// @audit requestL2Transaction\n37 contract MailboxFacet is Base, IMailbox {\n\n```\n*GitHub*: [[37](https://github.com/code-423n4/2023-10-zksync/blob/main/code/contracts/ethereum/contracts/zksync/facets/Mailbox.sol#L37-L37)]"
}
]
},
{
"title": "{{issue title}}",
"description": "{{issue description}}",
"instances": [
{
"link": "{{link to lines of code for judges to review}}",
"content": "{{content of issue, it can be combination of codes, text, diff and etc}}"
}
]
}
]
},
{
"severity": "M",
"issues": [...]
},
{
"severity": "G",
"issues": [...]
}
]
}
this can be used to produce:SummaryHigh Risk Issues
Total: 30 instances over 2 issues Medium Risk Issues{{description}}
{{extra information}} Gas OptimizationsGas totals use lower bounds of ranges and count two iterations of each
Total: 50 instances over 3 issues with 300 gas saved High Risk Issues[H‑01] Issue 1Description of issue 1 File: code/contracts/ethereum/contracts/zksync/facets/Mailbox.sol
// @audit requestL2Transaction
37 contract MailboxFacet is Base, IMailbox {
GitHub: [37] [H‑02] {{issue title}}{{issue description}} {{content of issue, it can be combination of codes, text, diff and etc}} Medium Risk Issues[...] Gas Issues[...] Notes:
|
Beta Was this translation helpful? Give feedback.
-
Very good move! Please, let's not overcomplicate this too much; we can always expand the structure. I have something similar as an intermediate step in generating Markdown. {
"issue": "MissingCheckAddressZeroInInitializer",
"severity": "Low",
"gas": null,
"title": "Missing checks for address(0) in initializer",
"description": [
"Check for zero-address to avoid the risk of setting address(0) for state variables when contract is initialized."
],
"instances": [
{
"contract": "contracts\/Tokens\/Prime\/Prime.sol",
"startLine": 130,
"endLine": 167,
"diff": null,
"comment": null
},
{
"contract": "contracts\/Tokens\/Prime\/PrimeLiquidityProvider.sol",
"startLine": 90,
"endLine": 111,
"diff": null,
"comment": null
}
]
} In the |
Beta Was this translation helpful? Give feedback.
-
Just a note re: rollout timeline - given the active, ongoing discussion here (which has been very fruitful, btw!), I think we'll need to revise the proposed dates. Oct 23 might still be feasible, but I appreciate that all bot crews will need sufficient time to update their bots. It would be helpful to hear how much notice would be reasonable, i.e. 1 week, 2 weeks, etc. |
Beta Was this translation helpful? Give feedback.
-
I'd suggest a minimum of 2 weeks notice. We need some finalized guidance on the JSON structure and then time to implement it. |
Beta Was this translation helpful? Give feedback.
-
JSON SchemaAfter reviewing these suggestions above and implementing changes to the new process, the new expected JSON is here:
Once the winning report has been selected, a Markdown file will be generated only for the winning report. Testing environmentTo give bot crews sufficient time to test the new submission requirements, a testing environment will be made available starting next week. Caveat: I will be OOO until Oct 28th, so please log bugs or change requests and I will address them when I'm back in the office. New ship date: Nov 6In order to give everyone sufficient time to change their bots to match this expected output -- and to allow me time to address any bugs or change requests -- the new date for these changes going live is Monday November 6th. All audits that start on or after the 6th will require bot reports to be submitted using the above JSON format. |
Beta Was this translation helpful? Give feedback.
-
Will "instance count" be defined as For example, using your earlier example, is the instance count here
|
Beta Was this translation helpful? Give feedback.
-
I was going to write the same thing, I don't think loc should be an array. Each instance should have its own json object. In the example given, there should be three separate instances. With separate instances, you can freely include whatever you want in the content (such as comments, etc.). The current one suggests that we should do one instance for each contract, and it will be more reasonable to do multiple instances per contract. When I say contract, I mean the .sol file. To sum up, we should have a main node for issues and then individual instances, either with a key enabling grouping by contract, or further nesting per contract. {
"comment": null,
"footnote": null,
"findings": [
{
"severity": "Medium",
"title": "Title of issue/finding",
"description": "description for the issue found",
"gasSavings": null,
"category": null,
"instances": [
{
"contract": "contracts\/ERC20MultiDelegate.sol",
"content": "",
"loc": 0
},
{
"contract": "contracts\/ERC20MultiDelegate.sol",
"content": "",
"loc": 0
}
]
}
]
} OR: {
"comment": null,
"footnote": null,
"findings": [
{
"severity": "Medium",
"title": "Title of issue/finding",
"description": "description for the issue found",
"gasSavings": null,
"category": null,
"instances": [
{
"contract": "contracts\/ERC20.sol",
"items": [
{
"content": "",
"loc": 0
}
]
},
{
"contract": "contracts\/ERC20MultiDelegate.sol",
"items": [
{
"content": "",
"loc": 0
}
]
}
]
}
]
} |
Beta Was this translation helpful? Give feedback.
-
Based on the feedback received so far on the submission testing environment, we have compiled a list of issues to fix / improve and expect to have most of them addressed by Nov 1, at which point further testing can be done. We'll assess readiness and the proposed ship date (Nov 6) later this week and post an update here and in Discord to either confirm Nov 6 or postpone by a week. |
Beta Was this translation helpful? Give feedback.
-
I would really, really, really, really like a file-based submission method. As it is, a lot of people's browsers/VMs crash when trying to paste large blobs, and having to copy and paste seems sub-optimal |
Beta Was this translation helpful? Give feedback.
-
Hello Everyone, Based on further feedback and updates, we have decided to postpone ship date by a week to allow time to make these fixes/updates. The new release date will be Nov 13th at which time all bot races and qualifiers will expect the new json format to be submitted. Thank you all for your feedback and patience. |
Beta Was this translation helpful? Give feedback.
-
Currently, bot reports are submitted as Markdown files, which are then parsed by the bot judges through a somewhat labor-intensive process (albeit with the assistance of some tools).
In the interest of reducing manual effort from judges, as well as furthering standardization to support better overall tooling, Code4rena bot races will soon require reports to be submitted in a standardized format.
New bot report format
Bot crews will submit a single, compilation-style bot report (as they currently do), but in .json format (i.e. no longer in Markdown).
Bot reports should be formatted like this:
Accepted severity levels are:
High
Medium
Low
Refactoring
NonCritical
Gas
Disputed
is also acceptable as a severity level, for validation purposes.The bot race submission form will be updated to validate that the correct format is used.
Proposed timeline
With these things in mind, we're looking for input as to whether October 16 or October 23, 2023 might be reasonable dates to implement this change -- meaning that bot races on or after the implementation date would require .json submissions.
Process notes
Beta Was this translation helpful? Give feedback.
All reactions