Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

core/types: remove support for legacy receipt/log storage encoding #22852

Merged
merged 2 commits into from
May 11, 2021

Conversation

fjl
Copy link
Contributor

@fjl fjl commented May 10, 2021

This removes the database upgrade path for receipts and logs stored before April 2019.

@fjl fjl added this to the 1.10.4 milestone May 10, 2021
@fjl fjl changed the title core/types: remove support for legacy storage encodings core/types: remove support for legacy receipt/log storage encodings May 10, 2021
@fjl fjl changed the title core/types: remove support for legacy receipt/log storage encodings core/types: remove support for legacy receipt/log storage encoding May 10, 2021
Copy link
Contributor

@holiman holiman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but let's hear what karalabe thinks too

Copy link
Member

@karalabe karalabe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Safe to delete as we introduced the freezer at the same time as the new format and the freezer moved everything reencoding, so nobody should have any old format stuff any more.

@karalabe karalabe merged commit 643fd0e into ethereum:master May 11, 2021
@shoenseiwaso
Copy link
Contributor

shoenseiwaso commented Jun 24, 2021

Note that this comment is intended only to help guide others who have the same problem, we are not expecting a fix.

For anyone who sees the following in their geth console logs with v1.10.4 or later:

ERROR[06-22|15:02:49.899] Invalid receipt array RLP                hash=28c41d..6bb895 err="rlp: expected input list for []*types.LogForStorage, decoding into ([]*types.ReceiptForStorage)[0](types.storedReceiptRLP).Logs"
ERROR[06-22|15:02:49.899] Invalid receipt array RLP                hash=506213..15eea1 err="rlp: expected input list for []*types.LogForStorage, decoding into ([]*types.ReceiptForStorage)[0](types.storedReceiptRLP).Logs"
ERROR[06-22|15:02:49.899] Invalid receipt array RLP                hash=fb8a03..3e976e err="rlp: expected input list for []*types.LogForStorage, decoding into ([]*types.ReceiptForStorage)[0](types.storedReceiptRLP).Logs"
ERROR[06-22|15:02:49.901] Invalid receipt array RLP                hash=506213..15eea1 err="rlp: expected input list for []*types.LogForStorage, decoding into ([]*types.ReceiptForStorage)[0](types.storedReceiptRLP).Logs"

Or sees the following error when they attempt to export their chaindata:

retrieved hash chain is invalid: invalid gas used (remote: 1884897 local: 1542885)

What probably happened is that at some point in the past, your geth database (chaindata) got silently corrupted and failed to automatically upgrade to the modern receipt/log storage encoding. With this change (#22852) go-ethereum removes the capability to read these legacy logs.

Possible solutions include:
(1) Recreate your geth database (chaindata) from scratch
(2) Postpone the upgrade to v1.10.4 or build a custom geth without this PR and run it until you can do (1)

fjl added a commit that referenced this pull request Jul 7, 2021
The encoding of Log and LogForStorage is exactly the same
now. After tracking it down it seems like #17106 changed the
storage schema of logs to be the same as the consensus
encoding.

Support for the legacy format was dropped in #22852 and if
I'm not wrong there's no reason anymore to have these two
equivalent types.

Since the RLP encoding simply contains the first three fields
of Log, we can also avoid creating a temporary struct for
encoding/decoding, and use the rlp:"-" tag in Log instead.

Note: this is an API change in core/types. We decided it's OK
to make this change because LogForStorage is an implementation
detail of go-ethereum and the type has zero uses outside of
package core/types.

Co-authored-by: Felix Lange <fjl@twurst.com>
sidhujag pushed a commit to sidhujag/go-ethereum that referenced this pull request Jul 10, 2021
The encoding of Log and LogForStorage is exactly the same
now. After tracking it down it seems like ethereum#17106 changed the
storage schema of logs to be the same as the consensus
encoding.

Support for the legacy format was dropped in ethereum#22852 and if
I'm not wrong there's no reason anymore to have these two
equivalent types.

Since the RLP encoding simply contains the first three fields
of Log, we can also avoid creating a temporary struct for
encoding/decoding, and use the rlp:"-" tag in Log instead.

Note: this is an API change in core/types. We decided it's OK
to make this change because LogForStorage is an implementation
detail of go-ethereum and the type has zero uses outside of
package core/types.

Co-authored-by: Felix Lange <fjl@twurst.com>
@rjl493456442
Copy link
Member

@shoenseiwaso not sure you are still around. Are you willing to recompile the geth with these diffs? I am curious why it's failed. Since we have the freezer to re-encode the legacy receipts. No legacy format receipts should be left.

diff --git a/core/rawdb/accessors_chain.go b/core/rawdb/accessors_chain.go
index 76132bf37..871ab0b64 100644
--- a/core/rawdb/accessors_chain.go
+++ b/core/rawdb/accessors_chain.go
@@ -572,7 +572,7 @@ func ReadRawReceipts(db ethdb.Reader, hash common.Hash, number uint64) types.Rec
        // Convert the receipts from their storage form to their internal representation
        storageReceipts := []*types.ReceiptForStorage{}
        if err := rlp.DecodeBytes(data, &storageReceipts); err != nil {
-               log.Error("Invalid receipt array RLP", "hash", hash, "err", err)
+               log.Error("Invalid receipt array RLP", "number", number, "hash", hash.Hex(), "err", err)
                return nil
        }
        receipts := make(types.Receipts, len(storageReceipts))

@rjl493456442
Copy link
Member

Let's discuss at #23245

rjl493456442 added a commit to rjl493456442/go-ethereum that referenced this pull request Jul 22, 2021
fjl pushed a commit that referenced this pull request Jul 22, 2021
* Revert "core/types: go generate (#23177)"

This reverts commit 00b922f.

* Revert "core/types: remove LogForStorage type (#23173)"

This reverts commit 7522642.

* Revert "core/types: remove support for legacy receipt/log storage encoding (#22852)"

This reverts commit 643fd0e.
sidhujag pushed a commit to sidhujag/go-ethereum that referenced this pull request Jul 28, 2021
* Revert "core/types: go generate (ethereum#23177)"

This reverts commit 00b922f.

* Revert "core/types: remove LogForStorage type (ethereum#23173)"

This reverts commit 7522642.

* Revert "core/types: remove support for legacy receipt/log storage encoding (ethereum#22852)"

This reverts commit 643fd0e.
atif-konasl pushed a commit to frozeman/pandora-execution-engine that referenced this pull request Oct 15, 2021
…thereum#22852)

* core/types: remove support for legacy receipt storage encoding

* core/types: remove support for legacy log storage encoding
atif-konasl pushed a commit to frozeman/pandora-execution-engine that referenced this pull request Oct 15, 2021
The encoding of Log and LogForStorage is exactly the same
now. After tracking it down it seems like ethereum#17106 changed the
storage schema of logs to be the same as the consensus
encoding.

Support for the legacy format was dropped in ethereum#22852 and if
I'm not wrong there's no reason anymore to have these two
equivalent types.

Since the RLP encoding simply contains the first three fields
of Log, we can also avoid creating a temporary struct for
encoding/decoding, and use the rlp:"-" tag in Log instead.

Note: this is an API change in core/types. We decided it's OK
to make this change because LogForStorage is an implementation
detail of go-ethereum and the type has zero uses outside of
package core/types.

Co-authored-by: Felix Lange <fjl@twurst.com>
atif-konasl pushed a commit to frozeman/pandora-execution-engine that referenced this pull request Oct 15, 2021
* Revert "core/types: go generate (ethereum#23177)"

This reverts commit 00b922f.

* Revert "core/types: remove LogForStorage type (ethereum#23173)"

This reverts commit 7522642.

* Revert "core/types: remove support for legacy receipt/log storage encoding (ethereum#22852)"

This reverts commit 643fd0e.
holiman pushed a commit that referenced this pull request Dec 3, 2022
This PR drops the legacy receipt types, the freezer-migrate command and the startup check. The previous attempt #22852 at this failed because there were users who still had legacy receipts in their db, so it had to be reverted #23247. Since then we added a command to migrate legacy dbs #24028.

As of the last hardforks all users either must have done the migration, or used the --ignore-legacy-receipts flag which will stop working now.
shekhirin pushed a commit to shekhirin/go-ethereum that referenced this pull request Jun 6, 2023
This PR drops the legacy receipt types, the freezer-migrate command and the startup check. The previous attempt ethereum#22852 at this failed because there were users who still had legacy receipts in their db, so it had to be reverted ethereum#23247. Since then we added a command to migrate legacy dbs ethereum#24028.

As of the last hardforks all users either must have done the migration, or used the --ignore-legacy-receipts flag which will stop working now.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants