SpuriousDragon HF Support #791

holgerd77 · 2020-06-25T08:38:30Z

Part of #652

References

Old working branch Spurious Dragon Working Branch #147
EIP 607: Hardfork Meta: Spurious Dragon
Spurious Dragon - Ethereum Blog Article (with EIPs)

TODO

Byzantium HF Activation Switches

Byzantium Meta Issue: #209

Check that all EIPs from the respective newer HF get disabled/reverted, taking in the list from #161 here:

SpuriousDragon HF EIPs - Recap and Activation Switch Checks

Rather for preparing older HF integration, technically not needed.

EIP-155 (Simple replay attack protection), implementation in the Tx library Support spurious dragon and later ethereumjs-tx#132 with HF switch in place
EIP-160 (EXP cost increase), done by having the EXP gas cost changes integrated in the Common library
EIP-161 (State trie clearing), will leave for now easier to adopt on TangerineWhistle integration
EIP-170 (Contract code size limit), added 7ccb6c3 as a check, but this is rather for older HF support and won't affect the SpuriousDragon test output directly

Test Situation

Test run with HF activated and no changes applied:

1..1221
# tests 1221
# pass  1126
# fail  95

codecov · 2020-06-25T08:40:34Z

Codecov Report

Merging #791 into master will increase coverage by 0.25%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #791      +/-   ##
==========================================
+ Coverage   84.19%   84.44%   +0.25%     
==========================================
  Files          16       18       +2     
  Lines        1221     1241      +20     
  Branches      246      246              
==========================================
+ Hits         1028     1048      +20     
  Misses        125      125              
  Partials       68       68

Flag	Coverage Δ
#account	`92.85% <ø> (ø)`
#block	`80.18% <ø> (+0.20%)`	⬆️
#blockchain	`84.50% <ø> (ø)`
#common	`93.60% <ø> (ø)`
#ethash	`86.30% <ø> (+0.89%)`	⬆️
#tx	`94.16% <ø> (+0.13%)`	⬆️

Impacted Files	Coverage Δ
packages/block/src/index.ts	`100.00% <0.00%> (ø)`
packages/tx/src/index.ts	`100.00% <0.00%> (ø)`
packages/ethash/src/util.ts	`97.22% <0.00%> (+0.92%)`	⬆️
packages/block/src/header-from-rpc.ts	`87.50% <0.00%> (+20.83%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cf82c4e...2e9572f. Read the comment docs.

holgerd77 · 2020-06-25T11:19:47Z

Ok, this is actually very much doable. Not completely there yet but state tests are passing already. 😄 The TODO list from above has still to be completed.

Was just working on re-adding pre-Byzantium EIP-658 (Tx receipts) when running out of time. This is the change commit here and I came up with the following not yet working test code for the runBlock.js API tests:

async function runWithHf(hardfork) {
  const vm = setupVM({ hardfork: hardfork })
  const suite = setup(vm)
  
  const block = new Block(util.rlp.decode(suite.data.blocks[0].rlp))
  
  await setupPreConditions(suite.vm.stateManager._trie, suite.data)
  
  let res = await suite.p.runBlock({
    block,
    generate: true,
    skipBlockValidation: true,
  })
  return res
}
  
tape('should return correct HF receipts', async (t) => {
  const res = await runWithHf('byzantium')
  t.equal(res.receipts[0].status, 1, 'should return correct post-Byzantium receipt format')
    
  res = await runWithHf('spuriousDragon')
  console.log(res)
  //t.equal(res.receipts[0].stateRoot, 1, 'should return correct pre-Byzantium receipt format')
  
  t.end()
})

Feel free everyone to pick this up - didn't want to commit incomplete here - and generally continue working on this (SpuriousDragon support). Shouldn't be too much work left.

evertonfraga · 2020-06-25T12:00:15Z

packages/vm/package.json

-    "test:state:allForks": "npm run test:state -- --fork=Byzantium && npm run test:state -- --fork=Constantinople && npm run test:state -- --fork=Petersburg && npm run test:state -- --fork=Istanbul && npm run test:state -- --fork=MuirGlacier",
-    "test:state:selectedForks": "npm run test:state -- --fork=Petersburg && npm run test:state -- --fork=Istanbul && npm run test:state -- --fork=MuirGlacier",
+    "test:state:allForks": "npm run test:state -- --fork=SpuriousDragon && npm run test:state -- --fork=Byzantium && npm run test:state -- --fork=Constantinople && npm run test:state -- --fork=Petersburg && npm run test:state -- --fork=Istanbul && npm run test:state -- --fork=MuirGlacier",
+    "test:state:selectedForks": "npm run test:state -- --fork=Petersburg && npm run test:state -- --fork=SpuriousDragon",


State tests are our bottleneck, I was thinking in splitting state tests in two jobs, so instead of a 12min job, it could be two jobs of 6:30s each. Let me know if that would speed up your work @holgerd77, so I can implement it right into this PR

Thanks for the offer, not necessary here on this round, but definitely useful to split these up. We can do test:state:selectedLatestForks and test:state:selectedNonLatestForks to give this some (somewhat blurry) semantics for better orientation, rather that test:state:selectedForks1 and ...2 or something.

Just a first-shot idea, feel free to take on or not.

I've been thinking about an experiment re state tests, and that'd be to have the test runner accept multiple forks, prepare everything once and run each test for all the forks consecutively. It probably won't yield much of a speed-up, but might be worth trying out if the effort required isn't that high.

Can you re-add Istanbul and MuirGlacier to seletedForks state test since those are run by CI?

My suggestion was to leave this to Everton to readd (see comment below) along splitting the state test run CI command into two (for performance reasons). Could you live with that as well?

holgerd77 · 2020-06-26T08:30:42Z

packages/vm/lib/runBlock.ts

+        ...abstractTxReceipt,
+      } as PreByzantiumTxReceipt
+    }
+


Ok, @s1na @alcuadrado please have a look if you are satisfied with this implementation, I thought it is not worth to update the checkpointing mechanism here, since this is a feature (which we then doesn't fully provide for pre-Byzantium) rather than consensus critical code.

We can alternatively also just fill in an empty Buffer and avoid changing the StateManager interface (might generally be a useful addition, not sure though). 🤔

Some working note: just tested to run the blockchain tests on older HFs with node -r ts-node/register --stack-size=1500 ./tests/tester --blockchain --fork=Byzantium.

These are currently NOT working and throwing errors like invalid block stateRoot and invalid receiptTrie. I can confirm though that this is also happening on master so this is not induced by the changes here and rather need some separate investigation. @evertonfraga We should also run - I would suggest - 1 additional fork config on the blockchain tests on the nightly test runs.

The Istanbul tests are running ('node -r ts-node/register --stack-size=1500 ./tests/tester --blockchain --fork=Istanbul'). The older tests not running doesn't necessarily mean that this originates in one or several bugs in our code but can also have its origin on the tests side and eventually the older HF blockchain tests don't get accurately generated any more respectively have been at the state of the testssnapshot.

I would rather suggest not to do this investigation here but do this separately. Otherwise the scope here gets too big.

The receiptTrie field in the block header is a commitment to the tx receipts, so I wouldn't be surprised if they were incorrect. But I agree we can probably handle this separately because as you said it needs deeper changes. I'd suggest however till then to keep the StateManager interface unchanged and if the intermediate state root is incorrect anyway then just return a dummy value until we dig deeper.

holgerd77 · 2020-06-26T09:23:41Z

Had another check, the difficulty code in BlockHeader.canonicalDifficulty() is actually already HF-aware (down to Frontier 😄 ), so there was just EIP-658 missing which I now added, please have a look there if you are satisfied with this since this is somewhat of a stub implementation, see reasoning along the code comment.

Gave this a last rebase, this is now ready for review.

@evertonfraga For practical reasons and to see the spuriousDragon state test results here, I would suggest to leave the changed test:state:selectedForks "as is" for this PR and you can then give this another update (likely without SpuriousDragon again ?) on your task split.

holgerd77 · 2020-06-26T11:36:18Z

Rebased this on top of #787

s1na

Great PR overall and thanks for taking initiative on this Holger! Just had a few minor comments.

s1na · 2020-06-26T10:23:15Z

packages/vm/lib/evm/opFns.ts

@@ -401,9 +401,15 @@ export const handlers: { [k: string]: OpHandler } = {
    runState.stack.push(new BN(keccak256(code)))
  },
  RETURNDATASIZE: function (runState: RunState) {
+    if (!runState._common.gteHardfork('byzantium')) {


Just a side-note: we won't be needing these checks anymore after getOpcodesForHF supports all the forks.

Yup these can be removed as these changes are in master.

Yes, but this will be another PR.

Can you approve here again so that we can merge? Sina already did but I dismissed by doing the rebase. 😋

packages/vm/lib/evm/opcodes.ts

packages/vm/lib/runBlock.ts

s1na · 2020-06-26T11:45:14Z

packages/vm/lib/runBlock.ts

+        ...abstractTxReceipt,
+      } as PreByzantiumTxReceipt
+    }
+


The receiptTrie field in the block header is a commitment to the tx receipts, so I wouldn't be surprised if they were incorrect. But I agree we can probably handle this separately because as you said it needs deeper changes. I'd suggest however till then to keep the StateManager interface unchanged and if the intermediate state root is incorrect anyway then just return a dummy value until we dig deeper.

holgerd77 · 2020-06-26T12:56:25Z

This is now ready for re-review. //cc @s1na

packages/vm/lib/evm/evm.ts

packages/vm/tests/api/runBlock.js

packages/vm/tests/config.js

packages/vm/lib/evm/evm.ts

holgerd77 · 2020-06-30T07:33:21Z

@ryanio thanks for the branch update, @s1na (or @evertonfraga if still awake 😛 , but don't stay up for this), can one of you give this another look?

holgerd77 · 2020-06-30T08:38:33Z

Update: please wait with merging this in here, I will do some other reviews this morning and then do a rebase here on top.

holgerd77 · 2020-06-30T11:40:11Z

Ok, rebased this. This is now open for another look.

holgerd77 · 2020-06-30T11:40:57Z

(also feel free to directly merge in upon review)

s1na

Looks good to me, thanks Holger!

…VM and package.json

…cution

…ASIZE and RETURNDATACOPY

…14) and REVERT (EIP-140)

…ForHF functionality

… to runBlock.js

holgerd77 · 2020-07-01T20:20:39Z

Rebased this, this would need a re-approval now.

holgerd77 added the PR state: WIP label Jun 25, 2020

evertonfraga reviewed Jun 25, 2020

View reviewed changes

holgerd77 commented Jun 26, 2020

View reviewed changes

holgerd77 force-pushed the spuriousDragon branch from 703e806 to 0230d05 Compare June 26, 2020 09:17

holgerd77 added PR state: needs review and removed PR state: WIP labels Jun 26, 2020

holgerd77 requested review from ryanio, evertonfraga, s1na and alcuadrado June 26, 2020 09:18

holgerd77 force-pushed the spuriousDragon branch from 0230d05 to 2e1508d Compare June 26, 2020 11:30

s1na reviewed Jun 26, 2020

View reviewed changes

holgerd77 force-pushed the spuriousDragon branch from 2e1508d to 5f55c10 Compare June 26, 2020 12:55

jochem-brouwer reviewed Jun 26, 2020

View reviewed changes

packages/vm/lib/evm/evm.ts Outdated Show resolved Hide resolved

packages/vm/tests/api/runBlock.js Show resolved Hide resolved

packages/vm/tests/config.js Show resolved Hide resolved

jochem-brouwer reviewed Jun 26, 2020

View reviewed changes

packages/vm/tests/config.js Show resolved Hide resolved

holgerd77 force-pushed the spuriousDragon branch from 5f55c10 to 2616b99 Compare June 26, 2020 14:57

jochem-brouwer reviewed Jun 26, 2020

View reviewed changes

packages/vm/lib/evm/evm.ts Show resolved Hide resolved

holgerd77 force-pushed the spuriousDragon branch from 2616b99 to f3017fd Compare June 26, 2020 15:52

jochem-brouwer mentioned this pull request Jun 29, 2020

Group opcodes based on HF #543

Closed

holgerd77 force-pushed the spuriousDragon branch from a0160af to 36f3d4e Compare June 30, 2020 11:38

holgerd77 added the PR state: merge ready label Jun 30, 2020

s1na previously approved these changes Jul 1, 2020

View reviewed changes

holgerd77 added 6 commits July 1, 2020 22:19

vm > spuriousDragon: added HF and alias (EIP-158) support to tester, …

0ef4add

…VM and package.json

vm > spuriousDragon: added allowed Code size check to EVM message exe…

d6fcb8e

…cution

vm -> spuriousDragon: added EIP-211 Byzantium HF checks for RETURNDAT…

e91cbb4

…ASIZE and RETURNDATACOPY

vm -> spuriousDragon: added Byzantium HF checks for STATICCALL (EIP-2…

7b4bf93

…14) and REVERT (EIP-140)

vm -> spuriousDragon: added dedicated Byzantium HF list to getOpcodes…

024f5c9

…ForHF functionality

vm -> spuriousDragon: added pre-Byzantium tx receipt format (EIP-658)…

2e9572f

… to runBlock.js

holgerd77 dismissed s1na’s stale review via 2e9572f July 1, 2020 20:20

holgerd77 force-pushed the spuriousDragon branch from 36f3d4e to 2e9572f Compare July 1, 2020 20:20

jochem-brouwer approved these changes Jul 1, 2020

View reviewed changes

holgerd77 merged commit dacc853 into master Jul 2, 2020

holgerd77 deleted the spuriousDragon branch July 2, 2020 07:26

jochem-brouwer mentioned this pull request Jul 31, 2020

Backporting old HF rules #652

Closed

5 tasks

holgerd77 mentioned this pull request Aug 9, 2020

VM v5 Release Planning #681

Closed

27 tasks

holgerd77 added the package: vm label Sep 4, 2020

This comment was marked as abuse.

Sign in to view

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SpuriousDragon HF Support #791

SpuriousDragon HF Support #791

holgerd77 commented Jun 25, 2020 •

edited

Loading

codecov bot commented Jun 25, 2020 •

edited

Loading

holgerd77 commented Jun 25, 2020

evertonfraga Jun 25, 2020

holgerd77 Jun 26, 2020

s1na Jun 26, 2020

s1na Jun 30, 2020

holgerd77 Jun 30, 2020

holgerd77 Jun 26, 2020

holgerd77 Jun 26, 2020

holgerd77 Jun 26, 2020

s1na Jun 26, 2020

holgerd77 commented Jun 26, 2020

holgerd77 commented Jun 26, 2020

s1na left a comment

s1na Jun 26, 2020

jochem-brouwer Jul 1, 2020

holgerd77 Jul 1, 2020

s1na Jun 26, 2020

holgerd77 commented Jun 26, 2020

holgerd77 commented Jun 30, 2020

holgerd77 commented Jun 30, 2020

holgerd77 commented Jun 30, 2020

holgerd77 commented Jun 30, 2020

s1na left a comment

holgerd77 commented Jul 1, 2020

This comment was marked as abuse.

SpuriousDragon HF Support #791

SpuriousDragon HF Support #791

Conversation

holgerd77 commented Jun 25, 2020 • edited Loading

References

TODO

Byzantium HF Activation Switches

SpuriousDragon HF EIPs - Recap and Activation Switch Checks

Test Situation

codecov bot commented Jun 25, 2020 • edited Loading

Codecov Report

holgerd77 commented Jun 25, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holgerd77 commented Jun 26, 2020

holgerd77 commented Jun 26, 2020

s1na left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holgerd77 commented Jun 26, 2020

holgerd77 commented Jun 30, 2020

holgerd77 commented Jun 30, 2020

holgerd77 commented Jun 30, 2020

holgerd77 commented Jun 30, 2020

s1na left a comment

Choose a reason for hiding this comment

holgerd77 commented Jul 1, 2020

This comment was marked as abuse.

holgerd77 commented Jun 25, 2020 •

edited

Loading

codecov bot commented Jun 25, 2020 •

edited

Loading