Sharding[VM,Block,Transaction,Message] and sharding transaction/message execution logic #238

NIC619 · 2017-12-21T11:42:41Z

What was wrong?

With new transaction format and to introduce ENTRY_POINT address(b'ffff….ffff'), some logics in vm.execute_transaction needed to be changed.
Then CREATE2(#224), PAYGAS(#234) and PANIC opcodes can be built on top.

How was it fixed?

ShardingMessage, ShardingTransaction, ShardingVM
- ShardingTransaction.intrinsic_gas
- validate_sharding_transaction
Account Abstraction
- ENTRY_POINT
- generate_create2_contract_address

Cute Animal Picture

hwwhww · 2017-12-21T14:50:43Z

evm/vm/forks/sharding/blocks.py

+)
+
+
+class ShardingBlock(HomesteadBlock):


Note that it should be class Collation(BaseCollation), which means BaseCollation is a new RLP object.
Ref: #172

If you're using ShardingBlock for tests currently, I think it's okay to keep it for now and we change to BaseCollation in other future task. But I don't see ShardingBlock being used?

yeah, the name will be changed later but for now I keep it this way for simplicity.

I suppose with new transaction class, there would have to be new block class?
the new block class will be part of the ShardingVM

ShardingVM = ByzantiumVM.configure( name='ShardingVM', _block_class=ShardingBlock, )

I think it was discussed but I can't recall the specifics.

What do you think about just writing an entirely separate Shard class so that we aren't overloading the block based API's, trying to cram the concepts of collations and blocks into the same API. While the two have a lot of similarities, I think they are going to be sufficiently different that we'll need a separate abstraction.

I have some deeper thoughts on the subject that I'll try to write-up since I'm starting to get the feeling that we will need a Shard class which mirrors the Chain class and than we'll want a ShardVM which mirrors the VM class.

Agree that it's better to write a Shard class. But maybe it can be replaced later? My goal for this PR is to change Transaction related code. (sorry about the misleading ShardingBlock, I figured if I want to test the modification on Transaction I need to also set up VM and Block class for Sharding.)

hwwhww · 2017-12-21T14:52:30Z

evm/rlp/transactions.py

@@ -151,7 +151,7 @@ class BaseShardingTransaction(rlp.Serializable):
        ('shard_id', big_endian_int),
        ('target', address),
        ('data', binary),
-        ('start_gas', big_endian_int),
+        ('gas', big_endian_int),


Could you give an explanation for why change start_gas to gas?

start_gas is only used in GasMeter while other Transaction classes use gas. So I change it for uniformity, but I have no strong preference for either one.

I've been using the term gas in the other Transaction classes to be more inline with the conventions used elsewhere such as web3.py or the JSON-RPC spec. I could be convinced to change it if someone has a compelling argument.

hwwhww · 2017-12-21T15:00:21Z

evm/vm/forks/sharding/validation.py

+    # TODO:Update transaction validation logic in Sharding
+    # e.g. checking shard_id < SHARD_COUNT
+
+    validate_frontier_transaction(evm, transaction)


Add a comment for using validate_frontier_transaction but validate_homestead_transaction on purpose would be clearer. (validate_homestead_transaction doesn't apply for sharding)

pipermerriam · 2017-12-22T18:48:50Z

evm/rlp/transactions.py

@@ -160,3 +160,32 @@ class BaseShardingTransaction(rlp.Serializable):
    @property
    def hash(self):
        return keccak(rlp.encode(self))
+
+    @property
+    def intrensic_gas(self):


organization nitpick

This seems like it should be next to the get_intrensic_gas method.

pipermerriam · 2017-12-22T18:50:58Z

Please get tests passing (looks like a linting error?).

NIC619 · 2017-12-25T13:54:27Z

@pipermerriam @hwwhww I update the code along with PR detail. Please have a look, thanks.

pipermerriam

This is good to go after cleaning up the code duplication in ShardingMessage.__init__. Let me know if I missed something that makes calling super() not work.

pipermerriam · 2017-12-26T22:32:55Z

evm/vm/message.py

+            validate_canonical_address(sender, title="Message.sender")
+        self.sender = sender
+
+        validate_uint256(value, title="Message.value")


I think you can eliminate the majority of the body of this function by inserting a super call to the Messag.__init__ method and then only dealing with what looks like the only difference which is access_list here in this method.

ah yeah you are right, I thought they differ in how they handle to and sender but the effects are actually the same.

pipermerriam · 2017-12-26T22:33:22Z

evm/vm/message.py

+        validate_canonical_address(to, title="Message.to")
+        self.to = to
+
+        if sender != ENTRY_POINT:


This check shouldn't be required since the ENTRY_POINT is a valid canonical address (at least I think it is)

yeah ENTRY_POINT is a valid canonical address so we only validate if sender is not ENTRY_POINT

Silly me. I saw == even though it's clearly !=

pipermerriam

Looks good.

It's a bit unnerving for me to have the various transaction and message execution functions for shards untested but I assume that will come at some later point.

NIC619 · 2017-12-28T09:56:29Z

@pipermerriam I add tests for is_create and intrensic_gas. Will figure out how to test execute_transaction, apply_message and apply_create_message next. Or I can pick transaction and message execution and their tests out and put them in next PR if this PR is considered fat enough.

jannikluhn · 2017-12-28T11:31:00Z

evm/utils/address.py

+
+def generate_create2_contract_address(salt, code):
+    """
+    If contract is created by transaction, `salt` should be empty.


Shouldn't the salt be specified by the transaction somehow (maybe the first 32 bytes of data)? Otherwise one can't deploy the same contract twice, or am I missing something?

The salt could be transaction nonce if nonce is not abstracted away eventually. Or if it is, one could simply append data to code and treat it as salt.

jannikluhn · 2017-12-28T12:47:37Z

evm/vm/forks/sharding/validation.py

+    # to be ECDSA and checks signatures accordingly. So it's skipped for
+    # the reason that there won't be built-in signature scheme in Account
+    # Abstraction.
+    validate_frontier_transaction(evm, transaction)


validate_frontier_transaction checks the balance of transaction.sender which would fail for ShardingTransactions. It also ignores transaction.access_list. So I think we can't use this.

yeah you are right. I will update it. Thanks!

jannikluhn · 2017-12-28T13:43:51Z

evm/vm/forks/sharding/__init__.py

+    #
+    # 2) Apply the message to the VM.
+    #
+    if message.is_create:


Right now message.is_create is True even if transaction.code is empty (as in this case message.code is set to the contract code in line 70). So I think there's something wrong here, but I'm not sure what's the best way to fix it.

ah that's also true, it's my mistake. And I don't see a clear and simple way around this either. Perhaps have message.is_create set by execute_transaction and CREATE* opcodes?

Yeah, that sounds reasonable.

I'm not sure if I'm reading this right, but if what you're suggesting involves mutating the Message object I would like to find another way as I think Message should be treated as immutable.

That makes sense. How about setting the message.is_create flag during message initialization? Since we can already pass in create_address, we can

simply add a new is_create parameter
OR

tell by checking the value of create_address
OR

just replace create_address with is_create (since in Account Abstraction, instead of CREATE_CONTRACT_ADDRESS, to would be set as the address of the new contract. Setting storage_address to create_address would be the same as leaving it empty).

I'd say go for 2 or 3; keep it simple.

NIC619 · 2017-12-28T16:36:10Z

btw, @jannikluhn @pipermerriam I was thinking 1. adding nonce back to transaction format OR 2. completely abstract nonce away.
Since nonce abstraction is still under discussion, I keep it where it was(as part of account's data). So right now nonce is not checked in transaction validation and also not incremented either.

I would prefer 1 and if nonce is eventually going away we can simply remove the codes altogether.

pipermerriam · 2017-12-28T18:16:19Z

@NIC619 my comment about testing was not intended to mean that you needed to provide those tests now (though I don't mind if you do), just to say that those code paths are un-tested. Maybe just a single smoke test for very basic transaction execution to know that there are no obvious major broken parts? I assume that the JSON fixture based tests will be updated at some point and then we'll use those to test.

pipermerriam · 2017-12-28T18:17:43Z

evm/vm/forks/sharding/validation.py

+    # to be ECDSA and checks signatures accordingly.
+    # `validate_frontier_transaction` checks for the balance of transaction
+    # sender.
+    # Both of which are in confilct with Account Abstraction implementation.


I think this whole comment can be removed with the understanding that this is just a new validation function.

pipermerriam · 2017-12-28T18:19:26Z

evm/vm/message.py

+
+    @property
+    def is_create(self):
+        return self.code not in (None, b'')


Hrm, it'd be nice to have a single representation of empty. What do you think about changing message.code to always be a bytestring so that b'' is the only empty value and we can just return bool(self.code)

vbuterin · 2017-12-29T10:45:08Z

evm/rlp/transactions.py

+        Hook called during instantiation to ensure that all transaction
+        parameters pass validation rules.
+        """
+        if self.intrensic_gas > self.gas:


Correct spelling is "intrinsic".

@pipermerriam I will update the naming in sharding branch along with this PR and update the one in master branch with a new PR.

pipermerriam · 2017-12-29T17:32:24Z

We should wait for #244 to be merged as this will need to be refactored to the new architecture introduced in that PR.

pipermerriam

There seems to be one un-used function. Otherwise this looks solid.

pipermerriam · 2018-01-02T21:22:15Z

tests/core/message-object/test_message_object.py

@@ -42,6 +44,27 @@ def _create_message(gas=1,
    )


+def _create_sharding_message(


This doesn't appear to be used anywhere.

…new transaction format

…m Message to ShardMessage

…g validation logic

…hereum#244

NIC619 · 2018-01-06T05:50:02Z

@pipermerriam rebased on #250 and updated to new computation logic(#244). If everything looks fine I will go ahead and merge this.

NIC619 added eth2.0 PR state: WIP labels Dec 21, 2017

NIC619 self-assigned this Dec 21, 2017

NIC619 requested review from pipermerriam and hwwhww December 21, 2017 11:42

hwwhww requested changes Dec 21, 2017

View reviewed changes

jannikluhn mentioned this pull request Dec 22, 2017

Standard user account #230

Merged

pipermerriam approved these changes Dec 22, 2017

View reviewed changes

NIC619 changed the title ~~ShardingBlock, BaseShardingTrasaction and ShardingTrasaction~~ Sharding[Block,Transaction,Message] and sharding transaction/message execution logic Dec 25, 2017

NIC619 changed the title ~~Sharding[Block,Transaction,Message] and sharding transaction/message execution logic~~ Sharding[VM,Block,Transaction,Message] and sharding transaction/message execution logic Dec 25, 2017

hwwhww approved these changes Dec 26, 2017

View reviewed changes

pipermerriam requested changes Dec 26, 2017

View reviewed changes

pipermerriam approved these changes Dec 27, 2017

View reviewed changes

jannikluhn reviewed Dec 28, 2017

View reviewed changes

pipermerriam reviewed Dec 28, 2017

View reviewed changes

vbuterin reviewed Dec 29, 2017

View reviewed changes

pipermerriam approved these changes Dec 29, 2017

View reviewed changes

ethereum deleted a comment from hwwhww Jan 2, 2018

pipermerriam approved these changes Jan 2, 2018

View reviewed changes

NIC619 force-pushed the account_abstraction_ENTRY_POINT branch from 3c0c378 to b02b514 Compare January 4, 2018 00:20

pipermerriam force-pushed the sharding branch from 0b9e4fb to 62e4934 Compare January 4, 2018 19:31

NIC619 force-pushed the account_abstraction_ENTRY_POINT branch from b02b514 to 519afdc Compare January 5, 2018 09:10

NIC619 added 15 commits January 6, 2018 09:54

update ShardingTransaction, __init__ and validate_sharding_transaction

6231f21

add ENTRY_POINT and new generate_contract_address function

49b4b19

change transaction/message execution logics to adopt ENTRY_POINT and …

95f3898

…new transaction format

add ShardingMessage

7bc0cd4

add apply_sharding_create_message and flake8 fix

d138c48

replace dulplicated __init__ code with super and move access_list fro…

09854e9

…m Message to ShardMessage

fix ShardingMesage.is_create and add is_creeat test

e82809c

add ShardingTransaction.intrensic_gas test

a6abcdc

remove comment and replace validate_frontier_transaction with Shardin…

efd0f23

…g validation logic

change is_create property of ShardingMessage

88f4774

rename "intrensic" to "intrinsic"

22fbc72

remove unused _create_sharding_message

9c5775c

amend: missing renaming of 'intrensic'

8536cd5

amend: missing vm/forks/sharding/blocks.py file

980dddf

update sharding computation to new computation logic introduced in et…

2646a10

…hereum#244

NIC619 force-pushed the account_abstraction_ENTRY_POINT branch from 519afdc to 2646a10 Compare January 6, 2018 03:18

NIC619 merged commit 5e1f216 into ethereum:sharding Jan 7, 2018

NIC619 removed the PR state: WIP label Jan 8, 2018

hwwhww mentioned this pull request Jan 8, 2018

Add sharding transaction test #252

Merged

jannikluhn mentioned this pull request Jan 8, 2018

STUB: New transaction format #194

Closed

1 task

hwwhww mentioned this pull request Jan 9, 2018

Shard class and ShardVM class #256

Closed

		@@ -42,6 +44,27 @@ def _create_message(gas=1,
		)


		def _create_sharding_message(

Sharding[VM,Block,Transaction,Message] and sharding transaction/message execution logic #238

Sharding[VM,Block,Transaction,Message] and sharding transaction/message execution logic #238

Conversation

NIC619 commented Dec 21, 2017 • edited Loading

What was wrong?

How was it fixed?

Cute Animal Picture

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hwwhww Dec 21, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pipermerriam commented Dec 22, 2017

NIC619 commented Dec 25, 2017

pipermerriam left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pipermerriam left a comment

Choose a reason for hiding this comment

NIC619 commented Dec 28, 2017 • edited Loading

jannikluhn Dec 28, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NIC619 Dec 29, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NIC619 commented Dec 28, 2017

pipermerriam commented Dec 28, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pipermerriam commented Dec 29, 2017

pipermerriam left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NIC619 commented Jan 6, 2018

NIC619 commented Dec 21, 2017 •

edited

Loading

hwwhww Dec 21, 2017 •

edited

Loading

NIC619 commented Dec 28, 2017 •

edited

Loading

jannikluhn Dec 28, 2017 •

edited

Loading

NIC619 Dec 29, 2017 •

edited

Loading