-
Notifications
You must be signed in to change notification settings - Fork 785
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix EIP2929 #1124
Fix EIP2929 #1124
Conversation
vm: update StateManager interface
vm: remove storage maps from interpreter
vm: add some extra EIP2929 tests (WIP)
packages/vm/lib/evm/evm.ts
Outdated
@@ -272,6 +273,10 @@ export default class EVM { | |||
} | |||
} | |||
|
|||
if (this._vm._common.eips().includes(2929)) { | |||
this._state.addWarmAddress(message.to.buf) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note: this is in the wrong place, we should do this before the checkpoint in the new message (in case that the creation fails, then the address should /still/ be in the warm list)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There will likely be a reason why you nevertheless put this here along with giving a comment, but it is not obvious too me (too difficult for some reasons?). Could you clarify?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah sorry this is a bit weird. I realized somewhere around pushing that this might be wrong, checked it, and realized that it is indeed wrong. It is more of a reminder comment that I should fix it :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
😄
Codecov Report
Flags with carried forward coverage won't be shown. Click here to find out more. |
// call to address 0xFFFF..FF | ||
//const callFF = '6000808080806000195AF1' | ||
//await runCodeTest(callFF + '60003415601C57600080808080305AF15B00', 1, t) | ||
//await runCodeTest('3415601557' + callFF + '600080FD5B600080808080305A60005400', 1, t) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These tests are WIP tests as described in message with the PR.
Small note, for #1048, the VM side of logic is now extremely easy, we can first |
@jochem-brouwer great analysis and integration, thanks so much Jochem! 😄 👍 Yeah, I think for (1) it should be ok to just clear after a tx? 🤔 Maybe just a generic public method Short side note: I would have a slight preference for (2) Would think that the suggested solution should work on first thought. Will give additional comments here if I get some additional insights here. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Really nice and clean process of doing this transition here! 👍 Have left some comments.
isWarmAddress(address: Buffer): boolean { | ||
for (let i = this._accessedStorage.length; i >= 0; i--) { | ||
const currentMap = this._accessedStorage[this._accessedStorage.length - 1] | ||
if (currentMap.has(address.toString('hex'))) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If I read the code correctly there seems to be no reason for this reverse order traversing (?) and this could then be simplified to for (const currentMap of this._accessedStorage)
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I did this in reverse order since this is an optimization. Imagine that we are rather deep in our call stack, say 10 levels deep. If we now access a storage slot on the current address (assuming that this is the first time we are at this address) then we actually mark the slot as warm in the end of the array. If we traverse the array in normal order, we first have to check the 9 other maps before figuring out that the 10th map (at current address) has marked it as warm. (It is a small optimization)
const storageKey = slot.toString('hex') | ||
|
||
for (let i = this._accessedStorage.length; i >= 0; i--) { | ||
const currentMap = this._accessedStorage[this._accessedStorage.length - 1] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same here, loop can be simplified.
packages/vm/lib/evm/evm.ts
Outdated
@@ -272,6 +273,10 @@ export default class EVM { | |||
} | |||
} | |||
|
|||
if (this._vm._common.eips().includes(2929)) { | |||
this._state.addWarmAddress(message.to.buf) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There will likely be a reason why you nevertheless put this here along with giving a comment, but it is not obvious too me (too difficult for some reasons?). Could you clarify?
packages/vm/lib/runTx.ts
Outdated
for (let addressInt = 1; addressInt <= numPrecompiles; addressInt++) { | ||
// Check if precompile exists on the current Common of the Block | ||
if (getPrecompile(new Address(Buffer.from(addressInt)), block._common)) { | ||
state.addWarmAddress(new Address(Buffer.from(addressInt))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not so beautiful to re-create this address space for the precompiles, we already have the precompiles
const in precompiles/index.ts
with all the addresses available, and this is also already exported.
Can you please use this here for address creation? Then there is also no need to do this extra numPrecompiles
export.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should have left a comment here. This is for forward compatibility. Geths implementation only marks current active precompiles to be warm. If we do not implement something like I created here (I agree it is not beautiful currently) then we get a consensus bug, for instance if we call a BLS precompile. Geth would mark it as cold, while we mark it as warm address. (GetPrecompile checks if precompile is available at current HF)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah ok, thanks. Can you then either:
a) add an additional getPrecompiles(common: Comon)
function to precompiles/index.ts
just looping over precompiles
and return the active ones analog to getPrecompile()
? or
b) minimal solution: at least do the numPrecompiles
count dynamically by doing a precompiles.length + 1
in precompiles/index.ts
, so that we do not introduce a relatively error prone new static constant there
This whole logic really doesn't belong there and we should at some point move to Common
(not now though likely).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I will go for option (a). #727 is related here
packages/vm/lib/state/interface.ts
Outdated
addWarmAddress(address: Buffer): void | ||
isWarmAddress(address: Buffer): boolean | ||
addWarmStorage(address: Buffer, slot: Buffer): void | ||
isWarmStorage(address: Buffer, slot: Buffer): boolean |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These methods unfortunately can't be added here yet to be mandatory, since this would be a breaking change.
Just sitting here realizing that this gets - again - a bit trickier from the compatibility angle. Suggestion: we add these here as optional and then make it very clear in the code docs /release notes, that - if people want to run EIP2929 (and likely also 2930) functionality AND have their custom state manager implementation - they need to implement these methods.
Ug. Likely also always (?) a good advice for people to do their custom implementations by inheriting from the default state manager. Then this has a higher chance that a custom state manager will continue to work on such kind of updates.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep you are right. I only fear that if I remove this, then TypeScript is going to complain about possible non existent methods...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, that's what we have to live with and what the !
operator is for. Unfortunately no way around that we just can't do such an interface change along the road without a major VM release. 😕
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sadly we cannot use the ! on methods. (I tried). Now we have to resort to explicit casting or casting the state manager as any =(
const warmRead = runState._common.param('gasPrices', 'warmstorageread') | ||
const coldSload = runState._common.param('gasPrices', 'coldsload') | ||
|
||
// @ts-ignore Set Object is possibly 'undefined' | ||
if (runState.accessedStorage.has(address) && runState.accessedStorage.get(address).has(keyStr)) { | ||
if (runState.stateManager.isWarmStorage(address, key)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the @ts-ignore line from above can be removed.
const result = await vm.runTx({ tx }) | ||
|
||
st.ok(result.gasUsed.toNumber() == expectedGasUsed) | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's a cool test helper function here. 😄
vm: make code backwards compatible vm: fix EIP2929 on CREATE* operations
vm: active precompiles method fixes
I think I resolved all issues above. Am pretty confident in the implementation currently. Just some small remarks: (1) Before we join YoloV3, it makes sense to at least get all Berlin tests passing. The basics are now covered by some API tests, but I'm certain that I've missed some edge cases here. |
PR run fails but should be easy fix (will do ASAP):
|
@@ -44,11 +46,12 @@ export function accessStorageEIP2929(runState: RunState, key: Buffer, isSstore: | |||
const baseFee = !isSstore ? runState._common.param('gasPrices', 'sload') : 0 | |||
const address = runState.eei.getAddress().buf | |||
|
|||
const slotIsCold = !runState.stateManager.isWarmStorage(address, key) | |||
const slotIsCold = !(<EIP2929StateManager>runState.stateManager).isWarmedStorage(address, key) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice and thought-through solution, cool! 👍
//await runCodeTest('3415601557' + callFF + '600080FD5B600080808080305A60005400', 1, t) | ||
const callFF = '6000808080806000195AF1' | ||
// call address 0xFF..FF, now call same contract again, call 0xFF..FF again (it is now warm) | ||
await runCodeTest(callFF + '60003415601B57600080808080305AF15B00', 23909, t) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Out of interest: how did you generate these bytcode sequences? 🤔
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One of my (I guess) somewhat esoteric hobbies is writing raw EVM assembly. I do this from time to time, and getting more "fluent" in raw EVM 😄 It is very cool to figure out how you can create a minimal implementation which does what it should do. The minimal proxy contract, for instance, is also very interesting. As a side note: I hope that at some point we get a new gas golf contest. They allow raw assembly too 😄
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lol, I absolutely need more hobbies I can brag about
Maybe you can give a course at some PiT 😁 (far from fluent assembly writer)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh yeah - that sounds fun 😄 , I can definitely prepare something at some PiT 👍
@jochem-brouwer Have updated this branch since I needed the latest changes from #1125 for some experimentation, hope that's ok, don't forget to pull the branch changes (and if you didn't and have already done some substantial amount of work: just force-push, maybe announce here before though). |
Currently running the following test Can this be right? I generally have to say that I am not understanding the role of this } else if (baseFee !== undefined && baseFee > 0) {
console.log(runState._common.param('gasPrices', 'warmstorageread'))
console.log(baseFee)
runState.eei.useGas(
new BN(runState._common.param('gasPrices', 'warmstorageread') - baseFee),
'EIP-2929 -> warmstorageread'
)
} Why is this subtracted at all for
🤔 |
Update: ah, just seeing from the screenshot that this is double-applied and first added and then subtracted again, likely for reasons of implementation simplification, but actually a bit for the cost of readability. Hmm. And this might have side effects to due to these steps happening in two separate PiTs in the control flow. |
The test from above actually also has such a slightly different gasUsage: Actual: So just a difference of |
Hi @holgerd77, I also had to do a double take here when I saw that it is negative, but this is correct. In total, for a CALL to a warm account, we charge the WARM gas (100 gas). Internally, what we do is we take the warm gas and the base call gas value (700), and subtract these. So we get -600. Later on (or earlier, but doesnt really matter) we charge the 700 gas, so net we charge 100 gas, which is correct according to the EIP. |
I am currently setting up retesteth and geth to get a trace of this transaction, but geth seems to have changed how we should start it in "retesteth" mode, and I cannot find any docs how I should run it 😢 |
I think if we do this ping-pong "first subtract the base fee, then add it again" thing we might operate on the wrong values for things like |
I'm not really sure if I like adding these exceptions to the base fee. You are right that it /might/ mess up maxCallGas, but since this operation is done before calling maxCallGas it is correct. I think this approach is elegant. |
(Or well, I am assuming it is correct - might also not be the case of course since this could be the culprit 😅 ) |
I have the impression that is no exception or something but just the direct implementation of the EIP (there is just no basefee for these opcodes, right? Correct me if I am wrong). This otherwise doesn't play well with this specification from the EIP:
And for example for the CREATE opcode the code looks like this: // '0xf0' range - closures
// 0xf0: CREATE
[
0xf0,
async function (runState: RunState) {
if (runState.eei.isStatic()) {
trap(ERROR.STATIC_STATE_CHANGE)
}
const [value, offset, length] = runState.stack.popN(3)
subMemUsage(runState, offset, length)
let gasLimit = new BN(runState.eei.getGasLeft())
gasLimit = maxCallGas(gasLimit, runState.eei.getGasLeft(), runState)
let data = Buffer.alloc(0)
if (!length.isZero()) {
data = runState.memory.read(offset.toNumber(), length.toNumber())
}
accessAddressEIP2929(runState, runState.eei.getAddress())
const ret = await runState.eei.create(gasLimit, value, data)
runState.stack.push(ret)
},
] This seems to be wrong? 🤔 Or is it just enough to switch the order here? |
This indeed looks wrong. Weird that tests do not pick this up (it seems that it is not picked up). |
What we could maybe do here is not explicitly add these exceptions in code, but just mark these methods to use 0 gas in Common, and handle the base fee in the functions themselves? |
👍 That sounds like a good way to handle it. |
Yup - found the bug with help of the trace! Thanks to @winsvega It has indeed to do with the base gas! At some point, if we have less than 700 gas (base call fee) then we go OOG when we deduct the fee. However, we actually charge 100 gas, so we should not go out of gas. I will fix this with the method described above (base gas to 0). In retrospect, I don't know why I haven't realized this earlier..! |
vm: fix early OOG errors on 2929
Berlin tests pass, have just added MuirGlacier to CI to ensure that it still works. Ready for review! Thanks for all the help on this one, @holgerd77! 😄 |
31aba34
to
d67e8bd
Compare
@jochem-brouwer yeah, thanks for going so straightly through this. Again - as I wrote in the other thread - really take some moment if you need to rebalance your time, these kind of in-between tasks just do large re-shuffles one just can't really plan in. 🙂 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So great, all checks pass, congrats! 😄
LGTM
Feel free to merge once this fits for you.
'EIP-2929 -> coldaccountaccess' | ||
) | ||
} | ||
// Warm: (selfdestruct beneficiary address reads are not charged when warm) | ||
} else if (baseFee !== undefined && baseFee > 0) { | ||
} else if (chargeGas && !isSelfdestruct) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's a lot better than these implicit assumptions from before! 👍
@@ -940,6 +940,8 @@ export const handlers: Map<number, OpHandler> = new Map([ | |||
|
|||
const [value, offset, length] = runState.stack.popN(3) | |||
|
|||
accessAddressEIP2929(runState, runState.eei.getAddress(), false) | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just as some post-review question: is the order with subMemUsage()
relevant here? Since in all the other cases this is the other way around?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good question, and also a bit sloppy of me. But I just explicitly checked; the memory gas cost is deterministic and does not depend on the current gas available, so the order does not matter. I don't really want to trigger a new CI run, so I will merge here.
Thanks for the quick review @holgerd77 😄 |
Closes #1123
The problem with the current implementation of EIP2929 is that the warm storage slots / addresses are only tracked per internal message, not on the entire transaction as the EIP implies. This PR moves the warm storage tracker from the Interpreter to the StateManager. This is done, because we also have to bookkeep if an internal message reverts. If that is the case, then extra added warm slots in these reverted internal messages, are now considered cold for the remainder of the transaction, instead of warm.
Since we explicitly revert/commit/checkpoint on StateManager, this is the logical place for this cache.
I also added an extra rule here, which I don't recall having removed, which also adds any created contract to the warm address list.
However, there are some problems here: (which is why this PR is WIP)
(1) We also use this checkpoint/commit/revert mechanism when running a block. Currently, this means warm storages are tracked during the entire block, which we don't want. Fix? Maybe we can explicitly clear the slots at the end of a transaction, and at the start of a transaction we also require an empty starting point?
(2) There's also a problem with runCall/runCode: this does never checkpoint state manager, and thus the current cache is the empty list. The current logic actually expects that there is a Map/Set available. Fix: we can probably initialize the default value as the empty Map, this should work fine. If we combine this with the fix above this should be OK. This is also the reason why the API tests file for EIP2929, since this uses runCode.
(I have written this down to also sit down and figure out what my current problems are, but came along with some thoughts for fixes along the way, I think this be fine. We also use this "end of tx" logic for the touched accounts.)
Current tests added as API tests:
API tests to add:
These do of course not cover everything, but give a rough idea that the implementation is correct.