Skip to content
This repository has been archived by the owner on Aug 13, 2019. It is now read-only.

Testing framework #12

Merged
merged 18 commits into from
May 5, 2019
Merged

Testing framework #12

merged 18 commits into from
May 5, 2019

Conversation

austinabell
Copy link
Contributor

This is the testing framework for running ETH tests from https://github.com/ethereum/tests.

To test this, run git submodule update --init to pull test files and then to run only the state test suite run

go test ./tests/state_test.go ./tests/util.go ./tests/init.go ./tests/vm_test_util.go ./tests/state_test_util.go

(You can just run go test ./tests or go test ./... but that will run all of the existing tests and is slow)

Entry point into these tests are as follows:

func TestETHHomestead(t *testing.T) {
	fns, _ := filepath.Glob(filepath.Join(ethGeneralStateDir, "stHomesteadSpecific", "*"))
	runETHTests(t, fns)
}

Where the stHomesteadSpecific file path can be changed to any of the folders within ethereum/tests submodule.

Note: The Byzantium and EIP158 fork tests will be failing currently as it depends on state trie clearing functionality (and possibly some other changes made since the DAO fork) but all other tests should pass.

There is existing logic in the code for handling deleting empty objects in the trie, but it isn't handled exactly how the ethereum protocol expects. This will probably be fixed with the EIP158/161 implementation in ETC that is being completed.

austinabell and others added 17 commits April 30, 2019 10:53
@austinabell
Removed coverage file accidentally commited
8c4a269
@austinabell
Set up testing framework for eth tests
f8abf14
@noot @austinabell
go: initialized modules (#10)
ef8429c 
* init gomodules

* remove vendor

* update modules
@austinabell
Updated struct formatting for unmarshalling
abca3a9
@austinabell
Updated format of eth test struct and updated test files (were replac…
827a04e 
…ed with generated files)
@austinabell
Using ethereum tests submodule and updated framework for testing
6663319
@austinabell
Updated hashing of logs comparison and logging of tests
b4bd778
@austinabell
Removed error checking on state execution because expected error in s…
1bc3652 
…ome tests
@austinabell
Updated testing structure into subtests for better reporting and so t…
e6d3e78 
…hat all test cases are tested even when one fails
@austinabell
Changed subtest name from full filepath to just the json file
9d0239d
@austinabell
Restructured and added light documentation
d68d0df
@austinabell
Added homestead specific state test
9a7b6d6
@austinabell
Updated framework to be able to delete empty objects in trie for test…
34f8064 
…s>EIP158
@austinabell
Changed conditional for when state objects are deleted and removed lo…
d32514e 
…gging statement
@austinabell
Changed folder back to intended from testing
9b1ef1e
@austinabell
Changed conditional to not skip EIP158 fork tests
d4f9780
@austinabell
Added functionality to skip tests
f622487
GregTheGreek
GregTheGreek suggested changes May 2, 2019
View reviewed changes
tests/util.go Outdated
Post map[string][]stPostState `json:"post"`
}

// // GenesisAlloc specifies the initial state that is part of the genesis block.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can remove the commented out code

soc1c
soc1c approved these changes May 4, 2019
View reviewed changes
Copy link
Contributor

@soc1c soc1c left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@GregTheGreek
Copy link

@soc1c HE'S ALIVE

@soc1c soc1c requested a review from noot May 5, 2019 08:52
@noot
Copy link
Contributor

noot commented May 5, 2019

TestETHHomestead fails when I run the tests:

--- FAIL: TestETHHomestead (0.01s)
    --- FAIL: TestETHHomestead/contractCreationOOGdontLeaveEmptyContract.json (0.00s)
        --- FAIL: TestETHHomestead/contractCreationOOGdontLeaveEmptyContract.json/Byzantium/0 (0.00s)
            state_test.go:733: Post state root error. Expected: 35df33d2a146ff660bb837914781857715d1b8752371b2f3e0768f29dd484775 have: 11ad11c6c73f16022220dad1de2eb827aa3c04fa651f7a7b15231ed98efab9bc
    --- FAIL: TestETHHomestead/contractCreationOOGdontLeaveEmptyContractViaTransaction.json (0.00s)
        --- FAIL: TestETHHomestead/contractCreationOOGdontLeaveEmptyContractViaTransaction.json/Byzantium/0 (0.00s)
            state_test.go:733: Post state root error. Expected: 86631b9f3536a824efa412c3162885c8bbb1657e315b8dbbda6a24a8ae8ff868 have: bf80ccfd1a7e6bf9f0ab98b938595389fd7240f2206c33bade862e49c19c2248
    --- FAIL: TestETHHomestead/createContractViaContract.json (0.00s)
        --- FAIL: TestETHHomestead/createContractViaContract.json/Byzantium/0 (0.00s)
            state_test.go:733: Post state root error. Expected: 00a243435c3922719d07f434874d20e02a1971ca2bc6d6359bcc18f7e1e766f1 have: 49c4c18bd400d25e0905c026db96e0b4441d2dffe0a62d47d1e9354a4166794d
    --- FAIL: TestETHHomestead/createContractViaContractOOGInitCode.json (0.00s)
        --- FAIL: TestETHHomestead/createContractViaContractOOGInitCode.json/Byzantium/0 (0.00s)
            state_test.go:733: Post state root error. Expected: f079b93388a39a4c343c66a917a044fb0d91de079786366d0c6c3b01bee07e7b have: ef627373222f9f8a0c974af676a99f9a8c62a585e7b8935cec7a26a7029ae6f0
    --- FAIL: TestETHHomestead/createContractViaTransactionCost53000.json (0.00s)
        --- FAIL: TestETHHomestead/createContractViaTransactionCost53000.json/Byzantium/0 (0.00s)
            state_test.go:733: Post state root error. Expected: 01d55964b2d7eb43d050c308c5835cbc9d0d876a35fd855f063ca4fc424c4e28 have: 087b558ee1cfed6f35debfdc37ea5ca5db71243f59fee0d6b9a0f8c5235355e8
FAIL
FAIL	command-line-arguments	6.233s

is this expected behaviour? everything else looks good to me

@soc1c
Copy link
Contributor

soc1c commented May 5, 2019

@noot yes

@noot noot merged commit 0595796 into development May 5, 2019
@noot noot deleted the austin/testing branch May 5, 2019 12:02
noot pushed a commit that referenced this pull request Jun 6, 2019
@soc1c @noot
Stage Atlantis Release on Master (#47)
5c3df66 
* Removed research and moved to wiki (#2)

* changed import paths from ethereumproject to eth-classic (#4)

* changed import paths from ethereumproject to eth-classic

* add bn256 package; begin adding bn256add

* fix syntax errors; tests package passes

* fix import path

* change all import paths to eth-classic after forking needed repos

* go: initialized modules (#10)

* init gomodules

* remove vendor

* update modules

* Fixed dependency references to allow build to run without sputnikvm (#14)

* Testing framework (#12)

* Removed coverage file accidentally commited

* Set up testing framework for eth tests

* go: initialized modules (#10)

* init gomodules

* remove vendor

* update modules

* Updated struct formatting for unmarshalling

* Updated format of eth test struct and updated test files (were replaced with generated files)

* Using ethereum tests submodule and updated framework for testing

* Updated hashing of logs comparison and logging of tests

* Removed error checking on state execution because expected error in some tests

* Updated testing structure into subtests for better reporting and so that all test cases are tested even when one fails

* Changed subtest name from full filepath to just the json file

* Restructured and added light documentation

* Added homestead specific state test

* Updated framework to be able to delete empty objects in trie for tests>EIP158

* Changed conditional for when state objects are deleted and removed logging statement

* Changed folder back to intended from testing

* Changed conditional to not skip EIP158 fork tests

* Added functionality to skip tests

* Removed commented out code

* Create CODEOWNERS (#17)

* Config and refactor (#21)

* Refactor and add config for Atlantis chain config

* Typo fix

* Refactor jump table setup

* Changed fork config skipping functionality and removed unused configuration mappings

* Fixed null pointer error with test configs

* Added definitions of other blocks for correctly mapping gas table

* Implemented IsAtlantis interface for vm runtime

* Added fix for testing some fork subtests EIP150 and Frontier (#27)

* ci: create basic circle-ci config (#26)

* ci: create basic circle-ci config

* ci: lower parallelism

* ci: rename jobs

* ci: rename jobs

* EIP161, State Trie Clearing (#28)

* implemented EIP 161 logic

* implemented EIP 161 logic

* fixed bug

* no more segfaults

* Byzantium Tests Passing. Certain Homestead Failing

* Fixed implementation to pass (almost) all tests

* Updated testing framework to run all ETH directories

* Reimpl EIP161 SUICIDE and CALL edge cases

* Skip unimplemented functionality tests

* proper indentation

* Elizabeth/fix bindings test (#30)

* add go mods to bind package

* attempt to fix mods

* revert some dependencies

* fixed trimToImportPath for outside of go path (#33)

* fixed trimToImportPath for outside of go path

* is this how working_directory works?

* is this how working_directory works?

* is this how working_directory works?

* is THIS how working_directory works?

* is THIS how working_directory works?

* working_directory

* working_directory

* EIP 140 REVERT (#34)

* Implemented EIP 140 op code framework to be tested

* Added required parameters and moved protocol parameters into their own file

* Removed coverage file accidentally commited

* Set up testing framework for eth tests

* Updated struct formatting for unmarshalling

* Updated format of eth test struct and updated test files (were replaced with generated files)

* Using ethereum tests submodule and updated framework for testing

* Updated testing structure into subtests for better reporting and so that all test cases are tested even when one fails

* Fix merge error

* Removed indirect reference to ethereum/go-ethereum during rebase

* Implemented EIP 140 op code framework to be tested

* Added required parameters and moved protocol parameters into their own file

* Removed coverage file accidentally commited

* Set up testing framework for eth tests

* Updated struct formatting for unmarshalling

* Updated format of eth test struct and updated test files (were replaced with generated files)

* Using ethereum tests submodule and updated framework for testing

* Updated testing structure into subtests for better reporting and so that all test cases are tested even when one fails

* Fix merge error

* Removed indirect reference to ethereum/go-ethereum during rebase

* WIP fixed some implementations of REVERT

* Fixed more implementation details

* Skipped unrelated and unimplemented tests

* Implemented EIP 140 op code framework to be tested

* Added required parameters and moved protocol parameters into their own file

* Removed coverage file accidentally commited

* Set up testing framework for eth tests

* Updated struct formatting for unmarshalling

* Updated format of eth test struct and updated test files (were replaced with generated files)

* Using ethereum tests submodule and updated framework for testing

* Updated testing structure into subtests for better reporting and so that all test cases are tested even when one fails

* Fix merge error

* Removed indirect reference to ethereum/go-ethereum during rebase

* Implemented EIP 140 op code framework to be tested

* Added required parameters and moved protocol parameters into their own file

* Removed coverage file accidentally commited

* Set up testing framework for eth tests

* Updated struct formatting for unmarshalling

* Updated format of eth test struct and updated test files (were replaced with generated files)

* Using ethereum tests submodule and updated framework for testing

* Updated testing structure into subtests for better reporting and so that all test cases are tested even when one fails

* Fix merge error

* Removed indirect reference to ethereum/go-ethereum during rebase

* WIP fixed some implementations of REVERT

* Fixed more implementation details

* Skipped unrelated and unimplemented tests

* Revert modules changes from development

* Removed last gas cost variable used previously to save recalculation

* EIP 170, contract size limit (#23)

* added contract size limit

* added maxCodeSizeExceeded error

* fixed some tests failing due to not checking for creation of contract in the OR

* added IsAtlantis condition for max code size to apply. We now fail tests

* removed TestETHCodeSizeLimit

* EIP 100 Difficulty adjustment and testing (#36)

* Set up testing framework for difficulty

* Set up framework for testing difficulty

* Implemented EIP 100 and set up testing config

* Set up testing framework for difficulty

* Set up framework for testing difficulty

* Implemented EIP 100 and set up testing config

* Cleaned up and moved params to file

* Fixed usages of CalcDifficulty

* Moved parsing of hex or decimal strings functions to common package

* EIP 211 and refactor (#37)

* set up framework for returndata and refactor

* Refactored pc operation

* Refactored nil function jumptable lookups (why was it like that to begin with)

* Finished refactoring jump operations

* Implemented 211 functionality, but state tests not passing

* Fixed returndatasize

* Refactor memory and gas stack accesses

* Fixed a few small details for the implementation

* Fixed incorrect implementation of returndatacopy and revert edge case

* Removed unused instruction parameter since refactor

* Removed commented out printing from debugging

* EIP 196, 197, 198 (#24)

* implement bn256 precompiles

* implement bigModExp

* add comments and test

* run precompile tests

* skip failing tests for now

* add distinction between atlantis and pre-atlantis precompiles

* refactor precompiles to separate pre and post atlantis

* implement bn256 precompiles

* implement bigModExp

* add comments and test

* run precompile tests

* skip failing tests for now

* add distinction between atlantis and pre-atlantis precompiles

* refactor precompiles to separate pre and post atlantis

* fix ecrecover edge case

* implement bn256 precompiles

* implement bigModExp

* add comments and test

* run precompile tests

* skip failing tests for now

* add distinction between atlantis and pre-atlantis precompiles

* refactor precompiles to separate pre and post atlantis

* implement bn256 precompiles

* implement bigModExp

* add comments and test

* run precompile tests

* fix ecrecover edge case

* update go.mod

* remove skip for previously failing tests

* attempt to fix go.mod

* attempt to fix go.mod

* fix go.mod termiu version

* EIP 214 STATICCALL (#40)

* Cherry pick STATICALL commit

* Removed implemented skipped tests

* Removed implemented tests
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@soc1c soc1c soc1c approved these changes

@noot noot noot approved these changes

@GregTheGreek GregTheGreek GregTheGreek approved these changes

Assignees
No one assigned
Labels
PR: 1 Needs Review Type: Tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@austinabell @GregTheGreek @noot @soc1c