eth: add debug_storageRangeAt #14350

fjl · 2017-04-19T10:13:00Z

The new RPC method allows downloading storage in chunks.

debug_storageRangeAt takes parameters blockHash, txIndex, account, startKey, limit. The startKey applies to the hashed key (instead of the preimage as originally suggested in #3407) . The returned storage values contain the preimage if available. The returned object also contains the nextKey field, which will be non-null if there are more keys after the range that was returned.

Example:

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "debug_storageRangeAt",
  "params": [
    "0xced82545131cdb6253f183a62dc3eaf81259be5cc744767798bcfb95664dc32c", // blockHash
    4,                                                                    // txIndex
    "0xB4A5BB81C6962Fa87Af6886ABE59b3cc56ec44b8",                         // account
    "0x",                                                                 // startKey
    2                                                                     // limit
  ]
}
// Response:
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "storage": {
      "0x1ab0c6948a275349ae45a06aad66a8bd65ac18074615d53676c09b67809099e0": {
        "key": "0x405787fa12a823e0f2b7631cc41b3ba8828b3321ca811111fa75cd3aa3bb5ace",
        "value": "0x00000000000000000000009413739a6ef2d14b5433465dbeffafbc2f2d610551"
      },
      "0x1d3954ea05fac9c4f44f290d32ed11be13922e2c8162b3594bb3bb0405546e8e": {
        "key": "0x2a359015afbfbe3b36db751c88f03aed63209927e8cf9ca7f0b69158e29e28f5",
        "value": "0x0000000000000000000000000000000000000000000000000000000000000004"
      }
    },
    "nextKey": "0x290decd9548b62a8d60345a988386fc84ba6bc95484008f6362f93160ef3e563"
  }
}

// Continuing the download using nextKey as startKey:
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "debug_storageRangeAt",
  "params": [
    "0xced82545131cdb6253f183a62dc3eaf81259be5cc744767798bcfb95664dc32c",
    4,
    "0xB4A5BB81C6962Fa87Af6886ABE59b3cc56ec44b8",
    "0x290decd9548b62a8d60345a988386fc84ba6bc95484008f6362f93160ef3e563",
    100
  ]
}
// The response has nextKey == null because there aren't any more values:
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "storage": {
      "0x290decd9548b62a8d60345a988386fc84ba6bc95484008f6362f93160ef3e563": {
        "key": "0x0000000000000000000000000000000000000000000000000000000000000000",
        "value": "0x0000000000000000000000945e237bf1540eea146f5e830593d1d54c279616a9"
      },
      ... // 16 more entries elided
    },
    "nextKey": null
  }
}

We agreed on a preliminary spec in #3407. This supersedes that PR.
The earlier commits leading up to the new API add support for seeking in the trie iterator.

The key was constructed from nibbles, which isn't possible for all nodes. Remove the only use of Key in LightTrie by always retrying with the original key that was looked up.

Arachnid · 2017-04-19T13:37:27Z

trie/encoding.go

+// in the case of an odd number. All remaining nibbles (now an even number) fit properly
+// into the remaining bytes. Compact encoding is used for nodes stored on disk.
+
+func hexToCompact(hex []byte) []byte {


Could you define typedefs for hex and compact encoding?

No, because both types of key can be stored in shortNode.Key.

We could always use casts there. Typedefs would be helpful to avoid confusion elsewhere - though the function naming accomplishes some of that at least.

I don't want to change it in this PR. We can do it in another PR.

Arachnid · 2017-04-19T13:39:46Z

trie/iterator.go

 }

-// NewNodeIterator creates an post-order trie iterator.
-func NewNodeIterator(trie *Trie) NodeIterator {
+// newNodeIterator creates an post-order trie iterator.


preorder. :)

Arachnid · 2017-04-19T13:40:14Z

trie/iterator.go

-// NewNodeIterator creates an post-order trie iterator.
-func NewNodeIterator(trie *Trie) NodeIterator {
+// newNodeIterator creates an post-order trie iterator.
+func newNodeIterator(trie *Trie, start []byte) NodeIterator {


What was the motivation behind making this internal?

The motivation was to have a single constructor for each iterator. Please see the commit message.

Arachnid · 2017-04-19T13:42:07Z

trie/iterator.go

-			for parent.child++; parent.child < len(node.Children); parent.child++ {
-				child := node.Children[parent.child]
+			// Full node, move to the first non-nil child.
+			i := parent.index + 1


Why isn't this part of the loop initializer any longer?

Arachnid · 2017-04-19T13:43:12Z

trie/iterator_test.go

+	if err := checkIteratorOrder(testdata1[1:], it); err != nil {
+		t.Fatal(err)
+	}
+}


Can you add tests for seeking to the empty string, and for seeking to a key past the end?

No need to test empty string, all other tests use NodeIterator(nil) ;). I've added one that seeks past the end.

Arachnid · 2017-04-20T09:31:31Z

eth/api_test.go

+		state, _ = state.New(common.Hash{}, db)
+		addr     = common.Address{0x01}
+		keys     = []common.Hash{
+			common.HexToHash("340dd630ad21bf010b4e676dbfa9ba9a02175262d1fa356232cfde6cb5b47ef2"),


I assume these are the hashes of the keys in the array below? Could you add a note to that effect for anyone coming across this later?

karalabe

Minor issues mostly. Please check and consider whether to fix or not.

One potentially bigger issue I saw was in the state object deep copy. Please check hat part out in detail and explain why it works if it works.

karalabe · 2017-04-24T14:04:40Z

trie/iterator.go

 	}

 	// Continue iteration to the next child
-outer:


❤️ I hate these labels so much :P

karalabe · 2017-04-24T14:12:03Z

trie/iterator.go

 				node, err := it.trie.resolveHash(hash, nil, nil)
 				if err != nil {
-					return err
+					return it.stack[len(it.stack)-1], &parent.index, it.path, err


Why return the previous "item" if peeking at the next one fails?

It has to return something. The idea was that the iterator wouldn't advance if there was an error fetching the next item.

karalabe · 2017-04-24T14:18:59Z

trie/encoding.go

+// KEYBYTES encoding contains the actual key and nothing else. This encoding is the
+// input to most API functions.
+//
+// HEX encoding contains on byte for each nibble of the key and an optional trailing


karalabe · 2017-04-24T14:21:03Z

trie/encoding.go

+// the second-lowest encoding whether the node at the key is a value node. The low nibble
+// of the first byte is zero in the case of an even number of nibbles and the first nibble
+// in the case of an odd number. All remaining nibbles (now an even number) fit properly
+// into the remaining bytes. Compact encoding is used for nodes stored on disk.


Ah sweet, I've never seen this part of the code in 2 years :))

karalabe · 2017-04-24T14:31:04Z

trie/encoding_test.go

+			key:    []byte{0x12, 0x34, 0x56},
+			hexIn:  []byte{1, 2, 3, 4, 5, 6},
+			hexOut: []byte{1, 2, 3, 4, 5, 6, 16},
+		},


Could you add a test for encoding/decoding empty keys? The previous code had explicit checks whereas the new code does this implicitly by relying on the calculation returning an empty slice. There's even a slight API change, where the old code returned nil in hexToKeybytes whereas the new returns []byte{}. I'd be happy to see an explicit test that verifies this weird case too.

karalabe · 2017-04-24T14:32:05Z

trie/encoding_test.go

+		{hex: []byte{0, 15, 1, 12, 11, 8, 16 /*term*/}, compact: []byte{0x20, 0x0f, 0x1c, 0xb8}},
+		// odd length, terminator
+		{hex: []byte{15, 1, 12, 11, 8, 16 /*term*/}, compact: []byte{0x3f, 0x1c, 0xb8}},
+	}


Could you add a test for encoding/decoding empty keys?

karalabe · 2017-04-24T14:39:21Z

core/state/state_object.go

+	if self.trie != nil {
+		// A shallow copy makes the two tries independent.
+		cpy := *self.trie
+		stateObject.trie = &cpy


Isn't stateObject.trie a trie.SecureTrie? Because that one is a much fancier construct that won't "deep copy" so easily https://github.com/ethereum/go-ethereum/blob/master/trie/secure_trie.go#L44

Shallow copy support for SecureTrie was added in 710435b51, by yours truly.

Don't trust that guy.... he looks shady af ;)

karalabe · 2017-04-24T14:40:13Z

core/state/statedb.go

@@ -296,6 +296,17 @@ func (self *StateDB) GetState(a common.Address, b common.Hash) common.Hash {
 	return common.Hash{}
 }

+// StorageTrie returns an iterator over storage trie of an account.


Doesn't look like an iterator to me :P

karalabe · 2017-04-24T14:45:38Z

eth/api.go

-			}, nil
-		case *ethapi.JavascriptTracer:
-			return tracer.GetResult()
+			return nil, vm.Context{}, nil, fmt.Errorf("tx %x failed: %v", tx.Hash(), err)


Can this ever happen? If the tx is already in our chain, I think it's safe to say this won't ever fail. (Mentioning because you deleted a similar check at tx.AsMessage).

The around AsMessage is for deriving the sender. I don't want to remove this one because many more things can go wrong (including DB errors, etc.).

'encode' and 'decode' are meaningless because the code deals with three encodings. Document the encodings and give a name to each one.

Make it so each iterator has exactly one public constructor: - NodeIterators can be created through a method. - Iterators can be created through NewIterator on any NodeIterator.

The 'step' method is split into two parts, 'peek' and 'push'. peek returns the next state but doesn't make it current. The end of iteration was previously tracked by setting 'trie' to nil. End of iteration is now tracked using the 'iteratorEnd' error, which is slightly cleaner and requires less code.

karalabe

LGTM

trie: remove Key in MissingNodeError

a31d268

The key was constructed from nibbles, which isn't possible for all nodes. Remove the only use of Key in LightTrie by always retrying with the original key that was looked up.

fjl requested a review from Arachnid April 19, 2017 10:13

fjl mentioned this pull request Apr 19, 2017

eth, internal/web3ext: implement storageRangeAt #3407

Closed

fjl force-pushed the trie-iterator-skip-2 branch 2 times, most recently from 5fb8474 to e1678ce Compare April 19, 2017 13:23

Arachnid suggested changes Apr 19, 2017

View reviewed changes

fjl force-pushed the trie-iterator-skip-2 branch from 69e1404 to 2f13ce1 Compare April 19, 2017 22:02

Arachnid reviewed Apr 20, 2017

View reviewed changes

Arachnid approved these changes Apr 20, 2017

View reviewed changes

fjl force-pushed the trie-iterator-skip-2 branch from 43de846 to b22dfc8 Compare April 20, 2017 10:23

karalabe added this to the 1.6.1 milestone Apr 20, 2017

karalabe requested changes Apr 24, 2017

View reviewed changes

fjl added 4 commits April 25, 2017 02:14

trie: rework and document key encoding

f958d7d

'encode' and 'decode' are meaningless because the code deals with three encodings. Document the encodings and give a name to each one.

trie: clean up iterator constructors

a13e920

Make it so each iterator has exactly one public constructor: - NodeIterators can be created through a method. - Iterators can be created through NewIterator on any NodeIterator.

eth: add debug_storageRangeAt

207bd7d

fjl force-pushed the trie-iterator-skip-2 branch from b22dfc8 to 207bd7d Compare April 25, 2017 00:14

karalabe approved these changes Apr 25, 2017

View reviewed changes

karalabe merged commit ba3bcd1 into ethereum:master Apr 25, 2017

3esmit mentioned this pull request May 31, 2017

add debug interface to rpc for supporting Remix IDE debugger. openethereum/parity-ethereum#5727

Closed

cdetrio mentioned this pull request Jul 29, 2017

Feature request, RPC interface: debug_getStorageAt #3374

Closed

cdetrio mentioned this pull request Apr 21, 2018

testeth via RPC requirements ethereum/retesteth#5

Closed

enlight mentioned this pull request Sep 16, 2019

Add support for Remix debugging of EVM calls loomnetwork/loomchain#1482

Open

aftermathdigital mentioned this pull request Jul 5, 2021

Add "debug_storageRangeAt" JSON-RPC method support NomicFoundation/hardhat#1631

Open

KoningR mentioned this pull request Jul 28, 2023

Client/StateManager: storageRangeAt() RPC call / EVMStateManager Interface Extension ethereumjs/ethereumjs-monorepo#2922

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

eth: add debug_storageRangeAt #14350

eth: add debug_storageRangeAt #14350

fjl commented Apr 19, 2017 •

edited

Loading

Arachnid Apr 19, 2017

fjl Apr 19, 2017

Arachnid Apr 19, 2017

fjl Apr 19, 2017

Arachnid Apr 19, 2017

Arachnid Apr 19, 2017

fjl Apr 19, 2017

Arachnid Apr 19, 2017

fjl Apr 19, 2017

Arachnid Apr 19, 2017

fjl Apr 19, 2017 •

edited

Loading

Arachnid Apr 20, 2017

karalabe left a comment

karalabe Apr 24, 2017

karalabe Apr 24, 2017

fjl Apr 25, 2017

karalabe Apr 24, 2017

karalabe Apr 24, 2017

karalabe Apr 24, 2017

karalabe Apr 24, 2017

karalabe Apr 24, 2017

fjl Apr 24, 2017 •

edited

Loading

karalabe Apr 25, 2017

karalabe Apr 24, 2017

karalabe Apr 24, 2017

fjl Apr 24, 2017

karalabe left a comment

eth: add debug_storageRangeAt #14350

eth: add debug_storageRangeAt #14350

Conversation

fjl commented Apr 19, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fjl Apr 19, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karalabe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fjl Apr 24, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karalabe left a comment

Choose a reason for hiding this comment

fjl commented Apr 19, 2017 •

edited

Loading

fjl Apr 19, 2017 •

edited

Loading

fjl Apr 24, 2017 •

edited

Loading