"Rebase" to geth 1.7.3 and add status geth patches #492

divan · 2017-12-04T12:17:44Z

This PR codifies our changes to the go-ethereum upstream in the form of diffs (.patch file format) and serves as a prerequisite to change our approach to syncronization with geth codebase.

Patches need to be updated per each geth release
They are up to date, so anyone can inspect them and clearly see what exactly we add into geth codebase

It also updates geth dependency in vendor/ to match latest v.1.7.3 release.

extract previous geth commits from our forked version and codify them as patches
update patches to work with 1.7.3 geth release
apply patches and update vendor/ folder
write scripts (or just doc?) for automatically applying patches for future releases
cht.json now contains only bootnodes addresses. New CHT/BloomTrie checkpoints are still hardcoded into go-ethereum and updated via patches.

README for patches: https://github.com/status-im/status-go/blob/adf7895549037bea5cc287636acb2e1236c56b44/geth-patches/README.md

Closes #467

adambabik · 2017-12-05T12:48:06Z

geth-patches/0000-accounts-hd-keys.patch

@@ -0,0 +1,352 @@
+diff --git a/accounts/keystore/key.go b/accounts/keystore/key.go


Should we store geth patches here or in our fork of go-ethereum?

@adambabik where to store patches is an open question, but with this approach we don't need to keep "our fork" repo anymore. Patches will be applied from scratch on each release (I believe this process could be automated via simple scripting)

adambabik · 2017-12-05T12:50:05Z

geth/node/node.go

@@ -166,12 +142,7 @@ func activateEthService(stack *node.Node, config *params.NodeConfig) error {
 	ethConf.DatabaseCache = config.LightEthConfig.DatabaseCache

 	if err := stack.Register(func(ctx *node.ServiceContext) (node.Service, error) {
-		lightEth, err := les.New(ctx, &ethConf)
-		if err == nil {
-			updateCHT(lightEth, config)


This is pretty important for us as otherwise LES must be synced from the beginning. How is it handled now? LES/2 feature?

I have just read it in the PR description...

One way to test it is to run a full node with LES enabled and try to sync your local node using that full node as a peer. We have a full node in our cluster. Let me know if you don't have access.

divan · 2017-12-05T16:55:36Z

Related
ethereum/go-ethereum#15575
ethereum/go-ethereum#15482

divan · 2017-12-08T16:56:59Z

After upgrading one of the Status bootnodes to Geth 1.7.3, and testing this PR, I can see that synchronization on Ropsten is starting from trusted checkpoint, hardcoded in Geth 1.7.3 code (and light mode sync takes around 5 mins on my laptop). This release seems to have implemented CHT functionality and my assumption is that trusted checkpoint should be updated regularly. Holding this PR until this moment is clarified.

adambabik

Where are CHT checkpoints hardcoded now?

themue

It's a limited LGTM as I'm not yet deep enough into the individual patches. But I like the approach and the realisation.

A final LGTM should come by @adambabik or @tiabc .

divan · 2017-12-12T14:24:32Z

Where are CHT checkpoints hardcoded now?

@adambabik they're implemented as a patch, which will be updated and re-applied from time to time until we implement proper Trusted Checkpoints mechanism: https://github.com/status-im/status-go/pull/492/files#diff-baeffc2b5580436f546bac21a047f513

Also, make sure you've seen this README for geth-patches dir, it also a good subject for review: https://github.com/status-im/status-go/blob/adf7895549037bea5cc287636acb2e1236c56b44/geth-patches/README.md

adambabik · 2017-12-13T14:35:02Z

geth-patches/0006-latest-cht.patch

+ 		bloomTrieRoot: common.HexToHash("aca7d7c504d22737242effc3fdc604a762a0af9ced898036b5986c3a15220208"),
+ 	}
+
+	statusRopstenCheckpoint = trustedCheckpoint{


I assume it has been calculated recently, right? When testing locally how long does it take to sync to the most recent block?

Yes, I did it a couple of days ago and sync took 9-10 seconds on my laptop.

adambabik · 2017-12-13T14:39:04Z

geth-patches/README.md

+git clone https://github.com/ethereum/go-ethereum
+
+# update remote url to point to our fork repo
+ git remote set-url origin git@github.com:status-im/go-ethereum.git


We could also recommend adding git remote set-url upstream git@github.com:ethereum/go-ethereum.git so any further pulls from upstream could be done like git pull upstream release/1.7:master.

Great! Feel free to update README. I like the way you proposed — it's more explicit, but can't check it now from scratch, so let's update this after this PR is merged. Both options work, right?

It sounds good. I will make a PR when this one is merged. Also, I will test the whole flow shortly after this PR is merged as I have a use-case for a new patch.

tiabc

Please, let me review the changes before merging.

divan · 2018-01-08T18:56:26Z

This PR is needed to be merged asap as it's blocking other PRs. I'm setting a deadline for tomorrow (Jan 9th). @tiabc if you have a chance to review it today, it would be great.

divan added the in progress label Dec 4, 2017

divan self-assigned this Dec 4, 2017

divan mentioned this pull request Dec 4, 2017

Feature/rebase go ethereum 1.7.3 #467 #490

Closed

3 tasks

divan changed the title ~~Add geth-related changes as patches~~ "Rebase" to geth 1.7.3 and add status geth patches Dec 4, 2017

adambabik reviewed Dec 5, 2017

View reviewed changes

divan force-pushed the rebase/patches branch from 4224db6 to ea0e813 Compare December 11, 2017 12:17

adambabik reviewed Dec 12, 2017

View reviewed changes

themue approved these changes Dec 12, 2017

View reviewed changes

divan removed the in progress label Dec 13, 2017

adambabik approved these changes Dec 13, 2017

View reviewed changes

divan force-pushed the rebase/patches branch from adf7895 to 6f07f4f Compare December 13, 2017 17:20

divan added 14 commits December 14, 2017 09:56

Add patches to geth in patch format

6dc1845

Replaced patches for 1.7.2

0927c65

Update patches to match Geth 1.7.3

977e1e9

Updated vendored go-ethereum to 1.7.3

f9537f7

Apply status patcheds

f0a5f83

Apply whisper delivery patch

94ffac1

Add whisper delivery patch

f8e9d3e

Fix les patch and remove updateCHT

4d56a2d

Add patch to update to latest CHT trusted checkpoint

d94c7c4

Apply patch with latest CHT update

7ddb765

Remove number & cht root hash from cht.json

067e247

Fix paths in patches

e3b1766

Change paths in .patch files back

831cd82

Add README for patches

48b7804

divan added 6 commits December 14, 2017 09:56

Remove backup files

f811c97

Update README

cd54333

Update README for upstream

dbb8b1a

Update README

5f86095

Update README

d316208

Remov obsolete patch

a861c24

divan force-pushed the rebase/patches branch from 8255631 to a861c24 Compare December 14, 2017 08:56

tiabc suggested changes Dec 21, 2017

View reviewed changes

divan mentioned this pull request Jan 8, 2018

Use single codepath for sending transactions to a local and remote nodes #527

Merged

divan mentioned this pull request Jan 10, 2018

Refactoring: Introduce vendoring library #223

Closed

tiabc approved these changes Jan 11, 2018

View reviewed changes

adambabik mentioned this pull request Jan 11, 2018

Extend documentation about go-ethereum changes #233

Closed

divan merged commit f93cd81 into develop Jan 15, 2018

adambabik deleted the rebase/patches branch December 27, 2019 10:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"Rebase" to geth 1.7.3 and add status geth patches #492

"Rebase" to geth 1.7.3 and add status geth patches #492

divan commented Dec 4, 2017 •

edited

Loading

adambabik Dec 5, 2017

divan Dec 5, 2017

adambabik Dec 5, 2017

adambabik Dec 5, 2017 •

edited

Loading

divan commented Dec 5, 2017

divan commented Dec 8, 2017

adambabik left a comment

themue left a comment

divan commented Dec 12, 2017 •

edited

Loading

adambabik Dec 13, 2017

divan Dec 13, 2017

adambabik Dec 13, 2017

divan Dec 13, 2017

adambabik Dec 14, 2017

tiabc left a comment

divan commented Jan 8, 2018

		@@ -0,0 +1,352 @@
		diff --git a/accounts/keystore/key.go b/accounts/keystore/key.go

"Rebase" to geth 1.7.3 and add status geth patches #492

"Rebase" to geth 1.7.3 and add status geth patches #492

Conversation

divan commented Dec 4, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adambabik Dec 5, 2017 • edited Loading

Choose a reason for hiding this comment

divan commented Dec 5, 2017

divan commented Dec 8, 2017

adambabik left a comment

Choose a reason for hiding this comment

themue left a comment

Choose a reason for hiding this comment

divan commented Dec 12, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tiabc left a comment

Choose a reason for hiding this comment

divan commented Jan 8, 2018

divan commented Dec 4, 2017 •

edited

Loading

adambabik Dec 5, 2017 •

edited

Loading

divan commented Dec 12, 2017 •

edited

Loading