swarm: snapshot load improvement #18220
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zelig
suggested changes
Dec 1, 2018
zelig
approved these changes
Dec 1, 2018
51 tasks
nolash
suggested changes
Dec 3, 2018
p2p/simulations/network.go
Outdated
| // Prepare connection events counter. | ||
| allConnected := make(chan struct{}) // closed when all connections are established | ||
| eventLoopStarted := make(chan struct{}) // ensures that event loop is started before it is closed | ||
| done := make(chan struct{}) // ensures that the event loop goroutine is terminated |
There was a problem hiding this comment.
I had the impression we named these channels with postfix C. I found this contributes to clarity. Did we stop doing this?
There was a problem hiding this comment.
I did not know that suffix C is a hard requirement. This variables are local and very close in this function, so I saw no need to suffix them. Do you insist on adding the C?
There was a problem hiding this comment.
I don't insist, but I just always do :)
| select { | ||
| case e := <-events: | ||
| // Detect only connection events. | ||
| if e.Type != EventTypeConn { |
There was a problem hiding this comment.
Should we not disregard the Control events aswell?
There was a problem hiding this comment.
We can, but could you elaborate what would we protect agains?
There was a problem hiding this comment.
I've experienced that for connections in p2p layer there are/can be two conn events being sent, the first of which is a Control event and will be fired even though the node object at that time will have Conn.Up = false for that connection in its state.
There was a problem hiding this comment.
Yes, I see, I forgot about this Up = false events. Thank you, that is a great suggestion. I've updated the code.
| // Nodes are still not connected or have been disconnected. | ||
| if !e.Conn.Up { | ||
| // Delete the connection from the set of established connections. | ||
| // This will prevent false positive in case disconnections happen. |
There was a problem hiding this comment.
Is this really correct? If disconnects happen,mustn't that mean we are not getting the snapshot we are expecting, since all connections that are not Up were pruned already from the snapshot.
There was a problem hiding this comment.
I did not experience any disconnections during snapshot loading, but please have a look at this conversation #18220 (comment).
There was a problem hiding this comment.
would be also nice to have a log.Warn here. this shouldn't be happening (I guess)
| } | ||
| // Check that the connection is from the snapshot. | ||
| for _, conn := range snap.Conns { | ||
| if conn.One == e.Conn.One && conn.Other == e.Conn.Other { |
There was a problem hiding this comment.
Are we entirely sure that this can't be the other way around when it comes back as events?
There was a problem hiding this comment.
I am pretty much sure. But please check the events code to verify it.
There was a problem hiding this comment.
@nolash i was thinking this too.
you could probably pull out the address comparison logic from p2p/simulations/network.go#ConnLabel to another function and use that before doing the comparison.
There was a problem hiding this comment.
@justelad interesting Idea about ConnLabel, but I think that the source/destination (One/Other) are intentionally called like that to specify the node which initiated the connection and the node which the connection was made to. Through the codebase that is an important distinction, which is also stated in p2p/simulations.Conn. Snapshots also keep this distinction in the json.
@nolash @justelad could you provide more details where this distinction breaks in the codebase?
There was a problem hiding this comment.
I don't know if it breaks. I asked. And you asked me to check. I haven't gotten around to checking yet, sorry! :)
There was a problem hiding this comment.
I can not find the breaking part and all snapshots from the stream package are loadable with exact checks on One/Other connection fields. It would be good to see if my assumption is correct. I did check and it would be good for someone else to check too, if there is a doubt about this check.
There was a problem hiding this comment.
I've looked through and I agree, the order of the conns will not change.
p2p/simulations/network.go
Outdated
|
|
||
| // Do not proceed until the goroutine with the event loop actually is ready | ||
| // to receive events. | ||
| <-eventLoopStarted |
There was a problem hiding this comment.
Doesn't the subscription start actually guarantee this already? When the subscription is executed, the channel queue will make sure all events come through (even block if not handled), no?
There was a problem hiding this comment.
No, it does not. Scheduling of two goroutines is not deterministic. Even subscription is done in a different goroutine. I thought that I covered explanations in the comments. Is there anything that you think that it needs more detailed elaboration in the code?
There was a problem hiding this comment.
The comments were clear, but I was under the assumption that subscribe would register the channel before returning, and thus generate a block on unread events. I wouldn't be surprised if I'm not the only one in our team with that assumption 🗡️
There was a problem hiding this comment.
I see. I pushed an alternative approach where the subscription is done in the main goroutine with unbuffered channel. This should ensure the required synchronization and it should be easier to read. I added a comment as events channel and subscription object are not used in the main goroutine.
There was a problem hiding this comment.
Ahh .. so the assumption was correct? I think I misunderstood what you wrote; when you said subscribe is in goroutine you meant your goroutine? 💃
There was a problem hiding this comment.
Yes, correct, the goroutine created in the Load method. I think that the updated code is simpler and easier to read, with the risk that someone will move the subscription to the (child) goroutine as it is only used in it, not in the main/parent goroutine. Because of this, I added a few words in the comment.
p2p/simulations/network.go
Outdated
| for _, conn := range snap.Conns { | ||
|
|
||
| if !net.GetNode(conn.One).Up || !net.GetNode(conn.Other).Up { | ||
| //in this case, at least one of the nodes of a connection is not up, | ||
| //so it would result in the snapshot `Load` to fail | ||
| continue | ||
| } | ||
| if err := net.Connect(conn.One, conn.Other); err != nil { | ||
| if err := net.Connect(conn.One, conn.Other); err != nil && !strings.Contains(err.Error(), "already connected") { |
There was a problem hiding this comment.
Why should "already connected" even be possible here?
There was a problem hiding this comment.
It is possible if discovery is set to true, which is not disabled in snapshot load function, but in the provided service. This change is protecting agains such scenario, which happened quite frequently when discovery is set to true or the fix NotifyPeer is not applied. I would keep ignoring this error as it may result in flaky tests if discovery is set to true.
There was a problem hiding this comment.
Isn't this in the end a bit of a catch 22 scenario? We want to make sure that the network is exactly as we had it in the snapshot, but if we boot with discovery this suggests it will never be. We should never pretend that it should. Maybe this warrants a bit more discussion?
There was a problem hiding this comment.
I agree that it needs more discussion. My reason for this approach is that the flakiness produced with exposing this error is not contributing to what is the core reason for already connected error. By reporting already connected error, we are requiring that discovery is set to false before the snapshot is loaded, and I am not sure if it is safe and correct to set it to true in the runtime, after the snapshot is loaded, if it needs to be enabled.
There was a problem hiding this comment.
I agree with @nolash. Why should we allow snapshot simulations to have discovery enabled in the first place?
There was a problem hiding this comment.
Cool, I will revert this change as discovery is a not needed.
p2p/simulations/network.go
Outdated
| // Wait until all connections from the snapshot are established. | ||
| case <-allConnected: | ||
| // Make sure that we do not wait forever. | ||
| case <-time.After(120 * time.Second): |
There was a problem hiding this comment.
I'm not so fond of these literals.
There was a problem hiding this comment.
A dedicated variable is added. I am not fond of introducing indirections just to avoid literals, but this one is very close.
| @@ -165,8 +165,8 @@ func (h *Hive) Run(p *BzzPeer) error { | |||
| // otherwise just send depth to new peer | |||
| dp.NotifyDepth(depth) | |||
| } | |||
| NotifyPeer(p.BzzAddr, h.Kademlia) | |||
There was a problem hiding this comment.
Could you please explain the effect of this change? never mind, I read the description last
acud
approved these changes
Dec 5, 2018
| } | ||
| // Check that the connection is from the snapshot. | ||
| for _, conn := range snap.Conns { | ||
| if conn.One == e.Conn.One && conn.Other == e.Conn.Other { |
There was a problem hiding this comment.
@nolash i was thinking this too.
you could probably pull out the address comparison logic from p2p/simulations/network.go#ConnLabel to another function and use that before doing the comparison.
| // Nodes are still not connected or have been disconnected. | ||
| if !e.Conn.Up { | ||
| // Delete the connection from the set of established connections. | ||
| // This will prevent false positive in case disconnections happen. |
There was a problem hiding this comment.
would be also nice to have a log.Warn here. this shouldn't be happening (I guess)
p2p/simulations/network.go
Outdated
| for _, conn := range snap.Conns { | ||
|
|
||
| if !net.GetNode(conn.One).Up || !net.GetNode(conn.Other).Up { | ||
| //in this case, at least one of the nodes of a connection is not up, | ||
| //so it would result in the snapshot `Load` to fail | ||
| continue | ||
| } | ||
| if err := net.Connect(conn.One, conn.Other); err != nil { | ||
| if err := net.Connect(conn.One, conn.Other); err != nil && !strings.Contains(err.Error(), "already connected") { |
There was a problem hiding this comment.
I agree with @nolash. Why should we allow snapshot simulations to have discovery enabled in the first place?
nolash
approved these changes
Dec 6, 2018
|
Thank you all for reviews and suggestions. |
TuitionCoin
added a commit
to FinTechToken/go-ethereum
that referenced
this pull request
Jun 16, 2019
* core: better printout of receipts in bad block reports (ethereum#18156) * core/blockchain: better printout of receipts in bad block reports * fix splleing * params: add Constantinople block to AllXYZProtocolChanges (ethereum#18162) * params: Add Constantinople block to AllCliqueProtocolChanges * params: Add Constantinople block to AllEthashProtocolChanges * trie: fix overflow in write cache parent tracking (ethereum#18165) trie/database: fix overflow in parent tracking * travis: increase open file limits (ethereum#18155) * cmd/swarm: FUSE do not require --ipcpath (ethereum#18112) - Have `${DataDir}/bzzd.ipc` as IPC path default. - Respect the `--datadir` flag. - Keep only the global `--ipcpath` flag and drop the local `--ipcpath` flag as flags might overwrite each other. (Note: before global `--ipcpath` was ignored even if it was set) fixes ethersphere#795 * eth: increase timeout in TestBroadcastBlock (ethereum#18064) * core: return error if repair block failed (ethereum#18126) * core: return error if repair block failed * make error a bit shorter * config: add constantinople block to testchainconfig * fix mixHash/nonce for parity compatible network (ethereum#18166) * les: fix fetcher syncing logic (ethereum#18072) * swarm/network/simulation: fix New function for-loop scope (ethereum#18161) * swarm/api: improve not found error msg (ethereum#18171) * light: odrTrie tryUpdate should use update (ethereum#18107) TryUpdate does not call t.trie.TryUpdate(key, value) and calls t.trie.TryDelete instead. The update operation simply deletes the corresponding entry, though it could retrieve later by odr. However, it adds further network overhead. * swarm/pss: Message handler refactor (ethereum#18169) * tests: Add flag to use EVMC for state tests (ethereum#18084) * Remove multihash from Swarm bzz:// for Feeds (ethereum#18175) * Accounting metrics reporter (ethereum#18136) * swarm/network: Correct neighborhood depth (ethereum#18066) * cmd/swarm: update should error on manifest mismatch (ethereum#18047) * cmd/swarm: fix ethersphere/swarm#979: update should error on manifest mistmatch * cmd/swarm: fixed comments and remove sprintf from log.Info * cmd/swarm: remove unnecessary comment * swarm: add database abstractions (shed package) (ethereum#18183) * Increase bzz version (ethereum#18184) * swarm/network/stream/: added stream protocol version match tests * Increase BZZ version due to streamer version change; version tests * swarm/network: increased hive and test protocol version * p2p/protocols: fix minor comments typo (ethereum#18185) * p2p/discv5: minor code simplification (ethereum#18188) * Update net.go more simple * Update net.go * p2p/discv5: gofmt * core: more detailed metrics for block processing (ethereum#18119) * eth/downloader: fix light client cht binary search issue * params: update CHTs for the v1.8.19 release * params, swarm: release Geth v1.8.19 and Swarm v0.3.7 * params, swarm: start Geth v1.8.20 and Swarm v0.3.8 release cycle * remove unrelated code * cmd/evm: commit statedb if dump is requested (ethereum#18208) Add a call `statedb.Commit(true)` if the `Dump` flag is on, as otherwise the `storage` output in the dump is always empty. * tests, core: update tests and make STATICCALL cause touch-delete (ethereum#18187) * vendor: update leveldb * remove a no-op line in the code (ethereum#17760) * mobile: added constructor for BigInts (ethereum#17828) * accounts/keystore: delete the redundant keystore in filename (ethereum#17930) * accounts/keystore: reduce file name length * accounts/keystore: reduce code line width * cmd/utils: max out the OS file allowance, don't cap to 2K * cmd/swarm: add flag for application name (swarm or swarm-private) (ethereum#18189) * cmd/swarm: add flag for application name (swarm or swarm-private) * cmd/swarm/swarm-smoke: return correct exit code * cmd/swarm/swarm-smoke: remove colorable * remove swarm/grafana_dashboards * vendor: update github.com/karalabe/hid (ethereum#18213) Fixes ethereum#15101 because hidapi is no longer being called from an init function. * p2p: use errors.New instead of fmt.Errorf (ethereum#18193) * swarm/pss: Add same api interface for all Send* methods (ethereum#18218) * signer/core: Fixes typo of method name in comment. (ethereum#18222) * Changed http:// to https:// on JSON-RPC link (ethereum#18224) Changed http:// to https:// on JSON-RPC link in README.md * cmd/puppeth: enforce lowercase network names * cmd/puppeth: implement chainspec converters * whisperv6: remove duplicated code (ethereum#18015) * cmd/puppeth: chain import/export via wizard, minor polishes * Add packing for dynamic array and slice types (ethereum#18051) * added tests for new abi encoding features (#4) * added tests from bytes32[][] and string[] * added offset to other types * formatting * Abi/dynamic types (#5) * Revert "Abi/dynamic types (#5)" (#6) This reverts commit dabca31. * Abi/dynamic types (#7) * some cleanup * Apply suggestions from code review apply suggestions Co-Authored-By: vedhavyas <vedhavyas.singareddi@gmail.com> * added better formatting (#8) * review chnages * better comments * swarm/network/stream: Debug log instead of Warn for retrieval failure (ethereum#18246) * swarm/api/http: add resetting timer metrics to requests (ethereum#18249) * core, internal, eth, miner, les: Take VM config from BlockChain (ethereum#17955) Until this commit, when sending an RPC request that called `NewEVM`, a blank `vm.Config` would be taken so as to set some options, based on the default configuration. If some extra configuration switches were passed to the blockchain, those would be ignored. This PR adds a function to get the config from the blockchain, and this is what is now used for RPC calls. Some subsequent changes need to be made, see ethereum#17955 (review) for the details of the discussion. * swarm: snapshot load improvement (ethereum#18220) * swarm/network: Hive - do not notify peer if discovery is disabled * p2p/simulations: validate all connections on loading a snapshot * p2p/simulations: track all connections in on snapshot loading * p2p/simulations: add snapshotLoadTimeout variable * p2p/simulations: ignore control events in snapshot load * p2p/simulations: simplify event loop synchronization * p2p/simulations: return already connected error from Load function * p2p/simulations: log warning on snapshot loading disconnection * node: warn when using deprecated config/resource files (ethereum#18199) * cmd/puppeth: fix rogue quote in alethGenesisSpec JSON (ethereum#18262) * eth/tracers: fixed incorrect storage from prestate_tracer (ethereum#18253) * eth: fixed incorrect storage from prestate_tracer * eth/tracers: updated assets.go * params: set mainnet and Rinkeby Constantinople fork blocks * cmd/evm, core/vm, eth: implement api methods to do stdjson dump to local filesystem * eth, internal/web3ext: tiny polishes in tracers * light: fix duplicated argument in bytes.Equal call Most probably a copy/paste kind of error. Found with gocritic `dupArg` checker. * ethereum: fix typo in interfaces.go (ethereum#18266) * Fix typo in interfaces.go * Update interfaces.go * cmd, eth: Add support for `--whitelist <blocknum>=<hash>,...` flag * Rejects peers that respond with a different hash for any of the passed in block numbers. * Meant for emergency situations when the network forks unexpectedly. * cmd/utils, eth: minor polishes on whitelist code * params: update CHTs for the 1.8.20 release * swarm/network: Correct ambiguity in compared addresses (ethereum#18251) * cmd/swarm, metrics, swarm/api/client, swarm/storage, swarm/metrics, swarm/api/http: add instrumentation (ethereum#18274) * cmd/faucet: fix faucet static peer regression * cmd/puppeth: support latest docker compose, expose faucet UDP * cmd/geth, core, eth: implement Constantinople override flag (ethereum#18273) * geth/core/eth: implement constantinople override flag * les: implemnent constantinople override flag for les clients * cmd/geth, eth, les: fix typo, move flag to experimentals * params, swarm: release Geth v1.8.20 and Swarm v0.3.8 * params, swarm: begin Geth v1.9.0 family, Swarm v0.3.9 cycle * swarm/shed: add metrics to each shed db (ethereum#18277) * swarm/shed: add metrics to each shed db * swarm/shed: push metrics prefix up * swarm/shed: rename prefix to metricsPrefix * swarm/shed: unexport Meter, remove Mutex for quit channel * swarm/storage: simplify ChunkValidator interface (ethereum#18285) * usbwallet: check returned error when decoding hexstr (ethereum#18056) * usbwallet: check returned error when decoding hexstr * Update accounts/usbwallet/ledger.go Co-Authored-By: CoreyLin <514971757@qq.com> * usbwallet: check hex decode error * crypto/secp256k1: Fix invalid document link (ethereum#18297) * accounts/abi: argument type and name were reversed (ethereum#17947) argument type and name were reversed * rpc: add application/json-rpc as accepted content type, fixes ethereum#18293 (ethereum#18310) * Comment error (ethereum#18303) * Change issue labels in bot configs to the new prefixed version * core/state: rename 'new' variable (ethereum#18301) * p2p/discv5: don't hash findnode target in lookup against table (ethereum#18309) * .github: add @gballet as abi codeowner (ethereum#18306) * fix slice unpack bug in accounts/abi (ethereum#18321) * fix slice unpack bug in accounts/abi * swarm/storage/feed: remove unused code (ethereum#18324) * p2p/simulation: move connection methods from swarm/network/simulation (ethereum#18323) * Update visualized snapshot test (ethereum#18286) * swarm/network/stream: fix visualized_snapshot_sync_sim_test * swarm/network/stream: updated visualized snapshot-test;data in p2p event * swarm/network/stream: cleanup visualized snapshot sync test * swarm/network/stream: re-enable t.Skip for visualized test * swarm/network/stream: addressed PR comments * swarm/network/simulation:commented out unreachable code-avoid vet errors (ethereum#18263) * swarm/pss: Reduce input vulnerabilities (ethereum#18304) * swarm/storage: remove unused methods from Chunk interface (ethereum#18283) * downloader: fix edgecase where returned index is OOB for downloader (ethereum#18335) * downloader: fix edgecase where returned index is OOB for downloader * downloader: documentation Co-Authored-By: holiman <martin@swende.se> * core: sanitize more TxPoolConfig fields (ethereum#17210) * core: sanitize more TxPoolConfig fields * core: fix TestTransactionPendingMinimumAllowance * p2p/simulation: Test snapshot correctness and minimal benchmark (ethereum#18287) * p2p/simulation: WIP minimal snapshot test * p2p/simulation: Add snapshot create, load and verify to snapshot test * build: add test tag for tests * p2p/simulations, build: Revert travis change, build test sym always * p2p/simulations: Add comments, timeout check on additional events * p2p/simulation: Add benchmark template for minimal peer protocol init * p2p/simulations: Remove unused code * p2p/simulation: Correct timer reset * p2p/simulations: Put snapshot check events in buffer and call blocking * p2p/simulations: TestSnapshot fail if Load function returns early * p2p/simulations: TestSnapshot wait for all connections before returning * p2p/simulation: Revert to before wait for snap load (5e75594) * p2p/simulations: add "conns after load" subtest to TestSnapshot and nudge * swarm/pss: forwarding function refactoring (ethereum#18353) * eth/downloader: progress in stateSync not used anymore (ethereum#17998) * p2p/protocols: accounting metrics rpc (ethereum#18336) * p2p/protocols: accounting metrics rpc added (ethereum#847) * p2p/protocols: accounting api documentation added (ethereum#847) * p2p/protocols: accounting api doc updated (ethereum#847) * p2p/protocols: accounting api doc update (ethereum#847) * p2p/protocols: accounting api doc update (ethereum#847) * p2p/protocols: fix file is not gofmted * fix lint error * updated comments after review * add account balance to rpc * naming changed after review * swarm/network: Revised depth and health for Kademlia (ethereum#18354) * swarm/network: Revised depth calculation with tests * swarm/network: WIP remove redundant "full" function * swarm/network: WIP peerpot refactor * swarm/network: Make test methods submethod of peerpot and embed kad * swarm/network: Remove commented out code * swarm/network: Rename health test functions * swarm/network: Too many n's * swarm/network: Change hive Healthy func to accept addresses * swarm/network: Add Healthy proxy method for api in hive * swarm/network: Skip failing test out of scope for PR * swarm/network: Skip all tests dependent on SuggestPeers * swarm/network: Remove commented code and useless kad Pof member * swarm/network: Remove more unused code, add counter on depth test errors * swarm/network: WIP Create Healthy assertion tests * swarm/network: Roll back health related methods receiver change * swarm/network: Hardwire network minproxbinsize in swarm sim * swarm/network: Rework Health test to strict Pending add test for saturation And add test for as many as possible up to saturation * swarm/network: Skip discovery tests (dependent on SuggestPeer) * swarm/network: Remove useless minProxBinSize in stream * swarm/network: Remove unnecessary testing.T param to assert health * swarm/network: Implement t.Helper() in checkHealth * swarm/network: Rename check back to assert now that we have helper magic * swarm/network: Revert WaitTillHealthy change (deferred to nxt PR) * swarm/network: Kademlia tests GotNN => ConnectNN * swarm/network: Renames and comments * swarm/network: Add comments * accounts/abi: add support for unpacking returned bytesN arrays (ethereum#15242) * accounts/abi: Brings out the msg defined at require statement in SC function (ethereum#17328) * swarm: remove unused/dead code (ethereum#18351) * fix string array unpack bug in accounts/abi (ethereum#18364) * core/types: update incorrect comment * accounts/abi: change unpacking of abi fields w/ underscores (ethereum#16513) * accounts/abi: fix name styling when unpacking abi fields w/ underscores ABI fields with underscores that are being unpacked into structs expect structs with following form: int_one -> Int_one whereas in abigen the generated structs are camelcased int_one -> IntOne so updated the unpack method to expect camelcased structs as well. * accounts/abi: fix case of generated java functions (ethereum#18372) * rpc: Warn the user when the path name is too long for the Unix ipc endpoint (ethereum#18330) * swarm/storage: change Proximity function and add TestProximity test (ethereum#18379) * build: add LGPL license at update-license.go (ethereum#18377) * add LGPL licence at update-licence.go * add empty line * travis, appveyor: bump to Go 1.11.4 (ethereum#18314) * travis, appveyor: bump to Go 1.11.4 * internal/build: revert comment changes * accounts/abi/bind: add optional block number for calls (ethereum#17942) * vendor: vendor/github.com/mattn/go-isatty - add missing files (reported by mksully22) (ethereum#18376) * swarm/docker: Dockerfile for swarm:edge docker image (ethereum#18386) * vendor, crypto, swarm: switch over to upstream sha3 package * vendor: update the entire golang.org/x/crypto dependency * cmd/puppeth: fix panic error when export aleth genesis wo/ precompile-addresses (ethereum#18344) * cmd/puppeth: fix panic error when export aleth genesis wo/ precompile-addresses * cmd/puppeth: don't need to handle duplicate set * eth/tracer: extend create2 (ethereum#18318) * eth/tracer: extend create2 * eth/tracers: fix create2-flaw in prestate_tracer * eth/tracers: fix test * eth/tracers: update assets * accounts/keystore: fix comment typo (ethereum#18395) * A few minor code inspection fixes (ethereum#18393) * swarm/network: fix code inspection problems - typos - redundant import alias * p2p/simulations: fix code inspection problems - typos - unused function parameters - redundant import alias - code style issue: snake case * swarm/network: fix unused method parameters inspections * accounts/keystore: small code simplification (ethereum#18394) * internal/ethapi: ask transaction pool for pending nonce (ethereum#15794) * cmd/geth: support dumpconfig optionally saving to file (ethereum#18327) * Changed dumpConfig function to optionally save to file * Added O_TRUNC flag to file open and cleaned up code * swarm: Shed Index and Uint64Field additions (ethereum#18398) * swarm/storage/mock/test: fix T.Fatal inside a goroutine (ethereum#18399) * swarm, p2p/protocols: Stream accounting (ethereum#18337) * swarm: completed 1st phase of swap accounting * swarm, p2p/protocols: added stream pricing * swarm/network/stream: gofmt simplify stream.go * swarm: fixed review comments * swarm: used snapshots for swap tests * swarm: custom retrieve for swap (less cascaded requests at any one time) * swarm: addressed PR comments * swarm: log output formatting * swarm: removed parallelism in swap tests * swarm: swap tests simplification * swarm: removed swap_test.go * swarm/network/stream: added prefix space for comments * swarm/network/stream: unit test for prices * swarm/network/stream: don't hardcode price * swarm/network/stream: fixed invalid price check * github: remove swarm github codeowners (ethereum#18412) * swarm: Fix T.Fatal inside a goroutine in tests (ethereum#18409) * swarm/storage: fix T.Fatal inside a goroutine * swarm/network/simulation: fix T.Fatal inside a goroutine * swarm/network/stream: fix T.Fatal inside a goroutine * swarm/network/simulation: consistent failures in TestPeerEventsTimeout * swarm/network/simulation: rename sendRunSignal to triggerSimulationRun * swarm/network: remove isproxbin bool from kad.Each* iterfunc (ethereum#18239) * swarm/network, swarm/pss: remove isproxbin bool from kad.Each* iterfunc * swarm/network: restore comment and unskip snapshot sync tests * accounts/abi: tuple support (ethereum#18406) * swarm/network: Rename minproxbinsize, add as member of simulation (ethereum#18408) * swarm/network: Rename minproxbinsize, add as member of simulation * swarm/network: Deactivate WaitTillHealthy, unreliable pending suggestpeer * accounts/abi: Extra slice tests (ethereum#18424) Co-authored-by: weimumu <934657014@qq.com> * p2p/simulations: eliminate concept of pivot (ethereum#18426) * core, eth: fix database version (ethereum#18429) * core, eth: fix database version * eth: polish error message * Stream subscriptions (ethereum#18355) * swarm/network: eachBin now starts at kaddepth for nn * swarm/network: fix Kademlia.EachBin * swarm/network: fix kademlia.EachBin * swarm/network: correct EachBin implementation according to requirements * swarm/network: less addresses simplified tests * swarm: calc kad depth outside loop in EachBin test * swarm/network: removed printResults * swarm/network: cleanup imports * swarm/network: remove kademlia.EachBin; fix RequestSubscriptions and add unit test * swarm/network/stream: address PR comments * swarm/network/stream: package-wide subscriptionFunc * swarm/network/stream: refactor to kad.EachConn * swarm/pot: pot.remove fixed (ethereum#18431) * swarm/pot: refactored pot.remove(), updated comments * swarm/pot: comments updated * swarm/pot: each() functions refactored (ethereum#18452) * accounts/abi: Add tests for reflection ahead of refactor (ethereum#18434) * params: postpone Constantinople due to net SSTORE reentrancy * params, swarm: release Geth v1.8.21 and Swarm v0.3.9 * core, cmd/puppeth: implement constantinople fix, disable EIP-1283 (ethereum#18486) This PR adds a new fork which disables EIP-1283. Internally it's called Petersburg, but the genesis/config field is ConstantinopleFix. The block numbers are: 7280000 for Constantinople on Mainnet 7280000 for ConstantinopleFix on Mainnet 4939394 for ConstantinopleFix on Ropsten 9999999 for ConstantinopleFix on Rinkeby (real number decided later) This PR also defaults to using the same ConstantinopleFix number as whatever Constantinople is set to. That is, it will default to mainnet behaviour if ConstantinopleFix is not set.This means that for private networks which have already transitioned to Constantinople, this PR will break the network unless ConstantinopleFix is explicitly set! * travis, appveyor: bump to Go 1.11.5 (ethereum#18947) * build: tweak debian source package build/upload options (ethereum#18962) dput --passive should make repo pushes from Travis work again. dput --no-upload-log works around an issue I had while uploading locally. debuild -d says that debuild shouldn't check for build dependencies when creating the source package. This option is needed to make builds work in environments where the installed Go version doesn't match the declared dependency in the source package. * p2p/discover, p2p/enode: rework endpoint proof handling, packet logging (ethereum#18963) This change resolves multiple issues around handling of endpoint proofs. The proof is now done separately for each IP and completing the proof requires a matching ping hash. Also remove waitping because it's equivalent to sleep. waitping was slightly more efficient, but that may cause issues with findnode if packets are reordered and the remote end sees findnode before pong. Logging of received packets was hitherto done after handling the packet, which meant that sent replies were logged before the packet that generated them. This change splits up packet handling into 'preverify' and 'handle'. The error from 'preverify' is logged, but 'handle' happens after the message is logged. This fixes the order. Packet logs now contain the node ID. * params: new CHTs (ethereum#18577) * p2p/discover: improve table addition code (ethereum#18974) This change clears up confusion around the two ways in which nodes can be added to the table. When a neighbors packet is received as a reply to findnode, the nodes contained in the reply are added as 'seen' entries if sufficient space is available. When a ping is received and the endpoint verification has taken place, the remote node is added as a 'verified' entry or moved to the front of the bucket if present. This also updates the node's IP address and port if they have changed. * params, swarm/version: Geth 1.8.22-stable, Swarm 0.3.10-stable * eth: make tracers respect pre- EIP 158/161 rule * core: fix error in block iterator (ethereum#18986) * cmd/puppeth: handle pre-set Petersburg number, save changed fork rules * core: repro ethereum#18977 * core: fix pruner panic when importing low-diff-large-sidechain * cmd, core, params: add support for Goerli (cherry picked from commit b0ed083) * build: use SFTP for launchpad uploads (ethereum#19037) * build: use sftp for launchpad uploads * .travis.yml: configure sftp export * build: update CI docs (cherry picked from commit 3de19c8) * common/fdlimit: cap on MacOS file limits, fixes ethereum#18994 (ethereum#19035) * common/fdlimit: cap on MacOS file limits, fixes ethereum#18994 * common/fdlimit: fix Maximum-check to respect OPEN_MAX * common/fdlimit: return error if OPEN_MAX is exceeded in Raise() * common/fdlimit: goimports * common/fdlimit: check value after setting fdlimit * common/fdlimit: make comment a bit more descriptive * cmd/utils: make fdlimit happy path a bit cleaner (cherry picked from commit f48da43) * .travis.yml: fix upload destination (ethereum#19043) (cherry picked from commit edf976e) * build: avoid dput and upload with sftp directly (ethereum#19067) (cherry picked from commit a8ddf7a) * common/fdlimit: fix windows build (ethereum#19068) (cherry picked from commit ba90a4a) * vendor: update syscalls dependency (cherry picked from commit dcc045f) * vendor: pull in upstream syscall fixes for non-linux/arm64 (cherry picked from commit 9d3ea8d) * trie: fix error in node decoding (ethereum#19111) * vendor: update bigcache (cherry picked from commit 37e5a90) * travis.yml: add launchpad SSH public key (ethereum#19115) (cherry picked from commit 75a9314) * build: explicitly force .xz compression (old debuild picks gzip) (ethereum#19118) (cherry picked from commit c0b9c76) * swarm/network: fix data race in TestNetworkID test (ethereum#18460) (cherry picked from commit 96c7c18) * swarm/storage: fix mockNetFetcher data races (ethereum#18462) fixes: ethersphere/swarm#1117 (cherry picked from commit f728837) * cmd/swarm/swarm-snapshot: swarm snapshot generator (ethereum#18453) * cmd/swarm/swarm-snapshot: add binary to create network snapshots * cmd/swarm/swarm-snapshot: refactor and extend tests * p2p/simulations: remove unused triggerChecks func and fix linter * internal/cmdtest: raise the timeout for killing TestCmd * cmd/swarm/swarm-snapshot: add more comments and other minor adjustments * cmd/swarm/swarm-snapshot: remove redundant check in createSnapshot * cmd/swarm/swarm-snapshot: change comment wording * p2p/simulations: revert Simulation.Run from master https://github.com/ethersphere/go-ethereum/pull/1077/files#r247078904 * cmd/swarm/swarm-snapshot: address pr comments * swarm/network/simulations/discovery: removed snapshot write to file * cmd/swarm/swarm-snapshot, swarm/network/simulations: removed redundant connection event check, fixed lint error (cherry picked from commit 34f11e7) * swarm/network: rewrite of peer suggestion engine, fix skipped tests (ethereum#18404) * swarm/network: fix skipped tests related to suggestPeer * swarm/network: rename depth to radius * swarm/network: uncomment assertHealth and improve comments * swarm/network: remove commented code * swarm/network: kademlia suggestPeer algo correction * swarm/network: kademlia suggest peer * simplify suggest Peer code * improve peer suggestion algo * add comments * kademlia testing improvements * assertHealth -> checkHealth (test helper) * testSuggestPeer -> checkSuggestPeer (test helper) * remove testSuggestPeerBug and TestKademliaCase * swarm/network: kademlia suggestPeer cleanup, improved comments * swarm/network: minor comment, discovery test default arg (cherry picked from commit bcb2594) * swarm/network: fix data race warning on TestBzzHandshakeLightNode (ethereum#18459) (cherry picked from commit 81e26d5) * Upload speed (ethereum#18442) (cherry picked from commit 257bfff) * cmd/swarm/swarm-snapshot: disable tests on windows (ethereum#18478) (cherry picked from commit 632135c) * cmd/swarm: use resetting timer to measure fetch time (ethereum#18474) (cherry picked from commit a0b0db6) * cmd/swarm/swarm-smoke: use ResettingTimer instead of Counters for times (ethereum#18479) (cherry picked from commit 5609577) * p2p/simulations: fix data race on swarm/network/simulations (ethereum#18464) (cherry picked from commit 85a79b3) * swarm: bootnode-mode, new bootnodes and no p2p package discovery (ethereum#18498) (cherry picked from commit bbd1203) * swarm: fix a data race on startTime (ethereum#18511) (cherry picked from commit fa34429) * swarm/version: commit version added (ethereum#18510) (cherry picked from commit ad13d2d) * p2p/testing, swarm: remove unused testing.T in protocol tester (ethereum#18500) (cherry picked from commit 2abeb35) * swarm/network: Remove extra random peer, connect test sanity, comments (ethereum#18964) (cherry picked from commit f9401ae) * swarm: fix flaky delivery tests (ethereum#18971) (cherry picked from commit 592bf6a) * swarm/storage: Get all chunk references for a given file (ethereum#19002) (cherry picked from commit 3eff652) * cmd/swarm: hashes command (ethereum#19008) (cherry picked from commit 7f55b0c) * cmd/swarm/swarm-smoke: sliding window test (ethereum#18967) (cherry picked from commit b91bf08) * cmd/swarm/swarm-smoke: remove wrong metrics (ethereum#18970) (cherry picked from commit c5c9cef) * cmd/swarm/swarm-smoke: refactor generateEndpoints (ethereum#19006) (cherry picked from commit d212535) * swarm: Debug API and HasChunks() API endpoint (ethereum#18980) (cherry picked from commit 41597c2) * swarm/storage: fix test timeout with -race by increasing mget timeout (cherry picked from commit 1c3aa8d) * swarm/pss: Remove pss service leak in test (ethereum#18992) (cherry picked from commit 7c60d0a) * swarm/storage/localstore: new localstore package (ethereum#19015) (cherry picked from commit 4f3d22f) * swarm/network: fix data race in fetcher_test.go (ethereum#18469) (cherry picked from commit 19bfcbf) * swarm/network, swarm/storage: Preserve opentracing contexts (ethereum#19022) (cherry picked from commit 0c10d37) * swarm/pss: transition to whisper v6 (ethereum#19023) (cherry picked from commit cde02e0) * swarm: GetPeerSubscriptions RPC (ethereum#18972) (cherry picked from commit 43e1b7b) * swarm/network: refactor simulation tests bootstrap (ethereum#18975) (cherry picked from commit 597597e) * cmd/swarm/global-store: global store cmd (ethereum#19014) (cherry picked from commit 33d0a0e) * swarm: CI race detector test adjustments (ethereum#19017) (cherry picked from commit 27e3f96) * swarm/docker: add global-store and split docker images (ethereum#19038) (cherry picked from commit 6cb7d52) * swarm/pss: mutex lifecycle fixed (ethereum#19045) (cherry picked from commit b30109d) * swarm/storage: fix HashExplore concurrency bug ethersphere#1211 (ethereum#19028) * swarm/storage: fix HashExplore concurrency bug ethersphere#1211 * swarm/storage: lock as value not pointer * swarm/storage: wait for to complete * swarm/storage: fix linter problems * swarm/storage: append to nil slice (cherry picked from commit 3d22a46) * swarm: fix uptime gauge update goroutine leak by introducing cleanup functions (ethereum#19040) (cherry picked from commit d596bea) * swarm: fix network/stream data races (ethereum#19051) * swarm/network/stream: newStreamerTester cleanup only if err is nil * swarm/network/stream: raise newStreamerTester waitForPeers timeout * swarm/network/stream: fix data races in GetPeerSubscriptions * swarm/storage: prevent data race on LDBStore.batchesC ethersphere/swarm#1198 (comment) * swarm/network/stream: fix TestGetSubscriptionsRPC data race ethersphere/swarm#1198 (comment) * swarm/network/stream: correctly use Simulation.Run callback ethersphere/swarm#1198 (comment) * swarm/network: protect addrCountC in Kademlia.AddrCountC function ethersphere/swarm#1198 (comment) * p2p/simulations: fix a deadlock calling getRandomNode with lock ethersphere/swarm#1198 (comment) * swarm/network/stream: terminate disconnect goruotines in tests * swarm/network/stream: reduce memory consumption when testing data races * swarm/network/stream: add watchDisconnections helper function * swarm/network/stream: add concurrent counter for tests * swarm/network/stream: rename race/norace test files and use const * swarm/network/stream: remove watchSim and its panic * swarm/network/stream: pass context in watchDisconnections * swarm/network/stream: add concurrent safe bool for watchDisconnections * swarm/storage: fix LDBStore.batchesC data race by not closing it (cherry picked from commit 3fd6db2) * swarm/storage/netstore: add fetcher cancellation on shutdown (ethereum#19049) swarm/network/stream: remove netstore internal wg swarm/network/stream: run individual tests with t.Run (cherry picked from commit 3ee09ba) * swarm/network: Saturation check for healthy networks (ethereum#19071) * swarm/network: new saturation for implementation * swarm/network: re-added saturation func in Kademlia as it is used elsewhere * swarm/network: saturation with higher MinBinSize * swarm/network: PeersPerBin with depth check * swarm/network: edited tests to pass new saturated check * swarm/network: minor fix saturated check * swarm/network/simulations/discovery: fixed renamed RPC call * swarm/network: renamed to isSaturated and returns bool * swarm/network: early depth check (cherry picked from commit 2af2472) * swarm/storage: fix influxdb gc metrics report (ethereum#19102) (cherry picked from commit 5b8ae78) * swarm/pss: refactoring (ethereum#19110) * swarm/pss: split pss and keystore * swarm/pss: moved whisper to keystore * swarm/pss: goimports fixed (cherry picked from commit 12ca3b1) * p2p, swarm: fix node up races by granular locking (ethereum#18976) * swarm/network: DRY out repeated giga comment I not necessarily agree with the way we wait for event propagation. But I truly disagree with having duplicated giga comments. * p2p/simulations: encapsulate Node.Up field so we avoid data races The Node.Up field was accessed concurrently without "proper" locking. There was a lock on Network and that was used sometimes to access the field. Other times the locking was missed and we had a data race. For example: ethereum#18464 The case above was solved, but there were still intermittent/hard to reproduce races. So let's solve the issue permanently. resolves: ethersphere/swarm#1146 * p2p/simulations: fix unmarshal of simulations.Node Making Node.Up field private in 13292ee broke TestHTTPNetwork and TestHTTPSnapshot. Because the default UnmarshalJSON does not handle unexported fields. Important: The fix is partial and not proper to my taste. But I cut scope as I think the fix may require a change to the current serialization format. New ticket: ethersphere/swarm#1177 * p2p/simulations: Add a sanity test case for Node.Config UnmarshalJSON * p2p/simulations: revert back to defer Unlock() pattern for Network It's a good patten to call `defer Unlock()` right after `Lock()` so (new) error cases won't miss to unlock. Let's get back to that pattern. The patten was abandoned in 85a79b3, while fixing a data race. That data race does not exist anymore, since the Node.Up field got hidden behind its own lock. * p2p/simulations: consistent naming for test providers Node.UnmarshalJSON * p2p/simulations: remove JSON annotation from private fields of Node As unexported fields are not serialized. * p2p/simulations: fix deadlock in Network.GetRandomDownNode() Problem: GetRandomDownNode() locks -> getDownNodeIDs() -> GetNodes() tries to lock -> deadlock On Network type, unexported functions must assume that `net.lock` is already acquired and should not call exported functions which might try to lock again. * p2p/simulations: ensure method conformity for Network Connect* methods were moved to p2p/simulations.Network from swarm/network/simulation. However these new methods did not follow the pattern of Network methods, i.e., all exported method locks the whole Network either for read or write. * p2p/simulations: fix deadlock during network shutdown `TestDiscoveryPersistenceSimulationSimAdapter` often got into deadlock. The execution was stuck on two locks, i.e, `Kademlia.lock` and `p2p/simulations.Network.lock`. Usually the test got stuck once in each 20 executions with high confidence. `Kademlia` was stuck in `Kademlia.EachAddr()` and `Network` in `Network.Stop()`. Solution: in `Network.Stop()` `net.lock` must be released before calling `node.Stop()` as stopping a node (somehow - I did not find the exact code path) causes `Network.InitConn()` to be called from `Kademlia.SuggestPeer()` and that blocks on `net.lock`. Related ticket: ethersphere/swarm#1223 * swarm/state: simplify if statement in DBStore.Put() * p2p/simulations: remove faulty godoc from private function The comment started with the wrong method name. The method is simple and self explanatory. Also, it's private. => Let's just remove the comment. (cherry picked from commit 50b872b) * cmd/swarm/swarm-smoke: Trigger chunk debug on timeout (ethereum#19101) * cmd/swarm/swarm-smoke: first version trigger has-chunks on timeout * cmd/swarm/swarm-smoke: finalize trigger to chunk debug * cmd/swarm/swarm-smoke: fixed httpEndpoint for trigger * cmd/swarm/swarm-smoke: port * cmd/swarm/swarm-smoke: ws not rpc * cmd/swarm/swarm-smoke: added debug output * cmd/swarm/swarm-smoke: addressed PR comments * cmd/swarm/swarm-smoke: renamed track-timeout and track-chunks (cherry picked from commit 62d7688) * contracts/*: golint updates for this or self warning (cherry picked from commit 53b823a) * swarm: Reinstate Pss Protocol add call through swarm service (ethereum#19117) * swarm: Reinstate Pss Protocol add call through swarm service * swarm: Even less self (cherry picked from commit d88c6ce) * swarm/metrics: Send the accounting registry to InfluxDB (ethereum#18470) (cherry picked from commit f28da4f) * cmd/utils: allow for multiple influxdb tags (ethereum#18520) This PR is replacing the metrics.influxdb.host.tag cmd-line flag with metrics.influxdb.tags - a comma-separated key/value tags, that are passed to the InfluxDB reporter, so that we can index measurements with multiple tags, and not just one host tag. This will be useful for Swarm, where we want to index measurements not just with the host tag, but also with bzzkey and git commit version (for long-running deployments). (cherry picked from commit 21acf0b) * params, swarm: release Geth v1.8.23, Swarm v0.3.11 * light, params: update CHTs, integrate CHT for Goerli too * common/fdlimit: fix macos file descriptors for Go 1.12 * all: simplify timestamps to uint64 (ethereum#19372) * all: simplify timestamps to uint64 * tests: update definitions * clef, faucet, mobile: leftover uint64 fixups * ethash: fix tests * graphql: update schema for timestamp * ethash: remove unused variable * trie: there's no point in retrieving the metaroot * travis: update builders to xenial to shadow Go releases * core: re-omit new log event when logs rebirth * core: minor code polishes + rebase fixes * params: set Rinkeby Petersburg fork block (4th May, 2019) * p2p/discover: bump failure counter only if no nodes were provided (ethereum#19362) This resolves a minor issue where neighbors responses containing less than 16 nodes would bump the failure counter, removing the node. One situation where this can happen is a private deployment where the total number of extant nodes is less than 16. Issue found by @Jsying. * eth, les, geth: implement cli-configurable global gas cap for RPC calls (ethereum#19401) * eth, les, geth: implement cli-configurable global gas cap for RPC calls * graphql, ethapi: place gas cap in DoCall * ethapi: reformat log message * params, swarm: release Geth v1.8.24 (noop Swarm 0.3.12) * cmd/geth: fix accidental --rpccorsdomain and --rpcvhosts removal * params, swarm: hotfix Geth v1.8.25 release to restore rpc flags * les: backported new SendTx cost calculation * les: check required message types in cost table * params, swarm: release Geth v1.8.26 (+noop Swarm v0.3.14) * eth, les, light: enforce CHT checkpoints on fast-sync too * params, swarm: release Geth v1.8.27 (noop Swarm v0.3.15) * forward to 1.8.27
TuitionCoin
added a commit
to FinTechToken/go-ethereum
that referenced
this pull request
Jun 16, 2019
* trie: fix overflow in write cache parent tracking (ethereum#18165) trie/database: fix overflow in parent tracking * travis: increase open file limits (ethereum#18155) * cmd/swarm: FUSE do not require --ipcpath (ethereum#18112) - Have `${DataDir}/bzzd.ipc` as IPC path default. - Respect the `--datadir` flag. - Keep only the global `--ipcpath` flag and drop the local `--ipcpath` flag as flags might overwrite each other. (Note: before global `--ipcpath` was ignored even if it was set) fixes ethersphere#795 * eth: increase timeout in TestBroadcastBlock (ethereum#18064) * core: return error if repair block failed (ethereum#18126) * core: return error if repair block failed * make error a bit shorter * config: add constantinople block to testchainconfig * fix mixHash/nonce for parity compatible network (ethereum#18166) * les: fix fetcher syncing logic (ethereum#18072) * swarm/network/simulation: fix New function for-loop scope (ethereum#18161) * swarm/api: improve not found error msg (ethereum#18171) * light: odrTrie tryUpdate should use update (ethereum#18107) TryUpdate does not call t.trie.TryUpdate(key, value) and calls t.trie.TryDelete instead. The update operation simply deletes the corresponding entry, though it could retrieve later by odr. However, it adds further network overhead. * swarm/pss: Message handler refactor (ethereum#18169) * tests: Add flag to use EVMC for state tests (ethereum#18084) * Remove multihash from Swarm bzz:// for Feeds (ethereum#18175) * Accounting metrics reporter (ethereum#18136) * swarm/network: Correct neighborhood depth (ethereum#18066) * cmd/swarm: update should error on manifest mismatch (ethereum#18047) * cmd/swarm: fix ethersphere/swarm#979: update should error on manifest mistmatch * cmd/swarm: fixed comments and remove sprintf from log.Info * cmd/swarm: remove unnecessary comment * swarm: add database abstractions (shed package) (ethereum#18183) * Increase bzz version (ethereum#18184) * swarm/network/stream/: added stream protocol version match tests * Increase BZZ version due to streamer version change; version tests * swarm/network: increased hive and test protocol version * p2p/protocols: fix minor comments typo (ethereum#18185) * p2p/discv5: minor code simplification (ethereum#18188) * Update net.go more simple * Update net.go * p2p/discv5: gofmt * core: more detailed metrics for block processing (ethereum#18119) * eth/downloader: fix light client cht binary search issue * params: update CHTs for the v1.8.19 release * params, swarm: release Geth v1.8.19 and Swarm v0.3.7 * params, swarm: start Geth v1.8.20 and Swarm v0.3.8 release cycle * remove unrelated code * cmd/evm: commit statedb if dump is requested (ethereum#18208) Add a call `statedb.Commit(true)` if the `Dump` flag is on, as otherwise the `storage` output in the dump is always empty. * tests, core: update tests and make STATICCALL cause touch-delete (ethereum#18187) * vendor: update leveldb * remove a no-op line in the code (ethereum#17760) * mobile: added constructor for BigInts (ethereum#17828) * accounts/keystore: delete the redundant keystore in filename (ethereum#17930) * accounts/keystore: reduce file name length * accounts/keystore: reduce code line width * cmd/utils: max out the OS file allowance, don't cap to 2K * cmd/swarm: add flag for application name (swarm or swarm-private) (ethereum#18189) * cmd/swarm: add flag for application name (swarm or swarm-private) * cmd/swarm/swarm-smoke: return correct exit code * cmd/swarm/swarm-smoke: remove colorable * remove swarm/grafana_dashboards * vendor: update github.com/karalabe/hid (ethereum#18213) Fixes ethereum#15101 because hidapi is no longer being called from an init function. * p2p: use errors.New instead of fmt.Errorf (ethereum#18193) * swarm/pss: Add same api interface for all Send* methods (ethereum#18218) * signer/core: Fixes typo of method name in comment. (ethereum#18222) * Changed http:// to https:// on JSON-RPC link (ethereum#18224) Changed http:// to https:// on JSON-RPC link in README.md * cmd/puppeth: enforce lowercase network names * cmd/puppeth: implement chainspec converters * whisperv6: remove duplicated code (ethereum#18015) * cmd/puppeth: chain import/export via wizard, minor polishes * Add packing for dynamic array and slice types (ethereum#18051) * added tests for new abi encoding features (#4) * added tests from bytes32[][] and string[] * added offset to other types * formatting * Abi/dynamic types (#5) * Revert "Abi/dynamic types (#5)" (#6) This reverts commit dabca31. * Abi/dynamic types (#7) * some cleanup * Apply suggestions from code review apply suggestions Co-Authored-By: vedhavyas <vedhavyas.singareddi@gmail.com> * added better formatting (#8) * review chnages * better comments * swarm/network/stream: Debug log instead of Warn for retrieval failure (ethereum#18246) * swarm/api/http: add resetting timer metrics to requests (ethereum#18249) * core, internal, eth, miner, les: Take VM config from BlockChain (ethereum#17955) Until this commit, when sending an RPC request that called `NewEVM`, a blank `vm.Config` would be taken so as to set some options, based on the default configuration. If some extra configuration switches were passed to the blockchain, those would be ignored. This PR adds a function to get the config from the blockchain, and this is what is now used for RPC calls. Some subsequent changes need to be made, see ethereum#17955 (review) for the details of the discussion. * swarm: snapshot load improvement (ethereum#18220) * swarm/network: Hive - do not notify peer if discovery is disabled * p2p/simulations: validate all connections on loading a snapshot * p2p/simulations: track all connections in on snapshot loading * p2p/simulations: add snapshotLoadTimeout variable * p2p/simulations: ignore control events in snapshot load * p2p/simulations: simplify event loop synchronization * p2p/simulations: return already connected error from Load function * p2p/simulations: log warning on snapshot loading disconnection * node: warn when using deprecated config/resource files (ethereum#18199) * cmd/puppeth: fix rogue quote in alethGenesisSpec JSON (ethereum#18262) * eth/tracers: fixed incorrect storage from prestate_tracer (ethereum#18253) * eth: fixed incorrect storage from prestate_tracer * eth/tracers: updated assets.go * params: set mainnet and Rinkeby Constantinople fork blocks * cmd/evm, core/vm, eth: implement api methods to do stdjson dump to local filesystem * eth, internal/web3ext: tiny polishes in tracers * light: fix duplicated argument in bytes.Equal call Most probably a copy/paste kind of error. Found with gocritic `dupArg` checker. * ethereum: fix typo in interfaces.go (ethereum#18266) * Fix typo in interfaces.go * Update interfaces.go * cmd, eth: Add support for `--whitelist <blocknum>=<hash>,...` flag * Rejects peers that respond with a different hash for any of the passed in block numbers. * Meant for emergency situations when the network forks unexpectedly. * cmd/utils, eth: minor polishes on whitelist code * params: update CHTs for the 1.8.20 release * swarm/network: Correct ambiguity in compared addresses (ethereum#18251) * cmd/swarm, metrics, swarm/api/client, swarm/storage, swarm/metrics, swarm/api/http: add instrumentation (ethereum#18274) * cmd/faucet: fix faucet static peer regression * cmd/puppeth: support latest docker compose, expose faucet UDP * cmd/geth, core, eth: implement Constantinople override flag (ethereum#18273) * geth/core/eth: implement constantinople override flag * les: implemnent constantinople override flag for les clients * cmd/geth, eth, les: fix typo, move flag to experimentals * params, swarm: release Geth v1.8.20 and Swarm v0.3.8 * params, swarm: begin Geth v1.9.0 family, Swarm v0.3.9 cycle * swarm/shed: add metrics to each shed db (ethereum#18277) * swarm/shed: add metrics to each shed db * swarm/shed: push metrics prefix up * swarm/shed: rename prefix to metricsPrefix * swarm/shed: unexport Meter, remove Mutex for quit channel * swarm/storage: simplify ChunkValidator interface (ethereum#18285) * usbwallet: check returned error when decoding hexstr (ethereum#18056) * usbwallet: check returned error when decoding hexstr * Update accounts/usbwallet/ledger.go Co-Authored-By: CoreyLin <514971757@qq.com> * usbwallet: check hex decode error * crypto/secp256k1: Fix invalid document link (ethereum#18297) * accounts/abi: argument type and name were reversed (ethereum#17947) argument type and name were reversed * rpc: add application/json-rpc as accepted content type, fixes ethereum#18293 (ethereum#18310) * Comment error (ethereum#18303) * Change issue labels in bot configs to the new prefixed version * core/state: rename 'new' variable (ethereum#18301) * p2p/discv5: don't hash findnode target in lookup against table (ethereum#18309) * .github: add @gballet as abi codeowner (ethereum#18306) * fix slice unpack bug in accounts/abi (ethereum#18321) * fix slice unpack bug in accounts/abi * swarm/storage/feed: remove unused code (ethereum#18324) * p2p/simulation: move connection methods from swarm/network/simulation (ethereum#18323) * Update visualized snapshot test (ethereum#18286) * swarm/network/stream: fix visualized_snapshot_sync_sim_test * swarm/network/stream: updated visualized snapshot-test;data in p2p event * swarm/network/stream: cleanup visualized snapshot sync test * swarm/network/stream: re-enable t.Skip for visualized test * swarm/network/stream: addressed PR comments * swarm/network/simulation:commented out unreachable code-avoid vet errors (ethereum#18263) * swarm/pss: Reduce input vulnerabilities (ethereum#18304) * swarm/storage: remove unused methods from Chunk interface (ethereum#18283) * downloader: fix edgecase where returned index is OOB for downloader (ethereum#18335) * downloader: fix edgecase where returned index is OOB for downloader * downloader: documentation Co-Authored-By: holiman <martin@swende.se> * core: sanitize more TxPoolConfig fields (ethereum#17210) * core: sanitize more TxPoolConfig fields * core: fix TestTransactionPendingMinimumAllowance * p2p/simulation: Test snapshot correctness and minimal benchmark (ethereum#18287) * p2p/simulation: WIP minimal snapshot test * p2p/simulation: Add snapshot create, load and verify to snapshot test * build: add test tag for tests * p2p/simulations, build: Revert travis change, build test sym always * p2p/simulations: Add comments, timeout check on additional events * p2p/simulation: Add benchmark template for minimal peer protocol init * p2p/simulations: Remove unused code * p2p/simulation: Correct timer reset * p2p/simulations: Put snapshot check events in buffer and call blocking * p2p/simulations: TestSnapshot fail if Load function returns early * p2p/simulations: TestSnapshot wait for all connections before returning * p2p/simulation: Revert to before wait for snap load (5e75594) * p2p/simulations: add "conns after load" subtest to TestSnapshot and nudge * swarm/pss: forwarding function refactoring (ethereum#18353) * eth/downloader: progress in stateSync not used anymore (ethereum#17998) * p2p/protocols: accounting metrics rpc (ethereum#18336) * p2p/protocols: accounting metrics rpc added (ethereum#847) * p2p/protocols: accounting api documentation added (ethereum#847) * p2p/protocols: accounting api doc updated (ethereum#847) * p2p/protocols: accounting api doc update (ethereum#847) * p2p/protocols: accounting api doc update (ethereum#847) * p2p/protocols: fix file is not gofmted * fix lint error * updated comments after review * add account balance to rpc * naming changed after review * swarm/network: Revised depth and health for Kademlia (ethereum#18354) * swarm/network: Revised depth calculation with tests * swarm/network: WIP remove redundant "full" function * swarm/network: WIP peerpot refactor * swarm/network: Make test methods submethod of peerpot and embed kad * swarm/network: Remove commented out code * swarm/network: Rename health test functions * swarm/network: Too many n's * swarm/network: Change hive Healthy func to accept addresses * swarm/network: Add Healthy proxy method for api in hive * swarm/network: Skip failing test out of scope for PR * swarm/network: Skip all tests dependent on SuggestPeers * swarm/network: Remove commented code and useless kad Pof member * swarm/network: Remove more unused code, add counter on depth test errors * swarm/network: WIP Create Healthy assertion tests * swarm/network: Roll back health related methods receiver change * swarm/network: Hardwire network minproxbinsize in swarm sim * swarm/network: Rework Health test to strict Pending add test for saturation And add test for as many as possible up to saturation * swarm/network: Skip discovery tests (dependent on SuggestPeer) * swarm/network: Remove useless minProxBinSize in stream * swarm/network: Remove unnecessary testing.T param to assert health * swarm/network: Implement t.Helper() in checkHealth * swarm/network: Rename check back to assert now that we have helper magic * swarm/network: Revert WaitTillHealthy change (deferred to nxt PR) * swarm/network: Kademlia tests GotNN => ConnectNN * swarm/network: Renames and comments * swarm/network: Add comments * accounts/abi: add support for unpacking returned bytesN arrays (ethereum#15242) * accounts/abi: Brings out the msg defined at require statement in SC function (ethereum#17328) * swarm: remove unused/dead code (ethereum#18351) * fix string array unpack bug in accounts/abi (ethereum#18364) * core/types: update incorrect comment * accounts/abi: change unpacking of abi fields w/ underscores (ethereum#16513) * accounts/abi: fix name styling when unpacking abi fields w/ underscores ABI fields with underscores that are being unpacked into structs expect structs with following form: int_one -> Int_one whereas in abigen the generated structs are camelcased int_one -> IntOne so updated the unpack method to expect camelcased structs as well. * accounts/abi: fix case of generated java functions (ethereum#18372) * rpc: Warn the user when the path name is too long for the Unix ipc endpoint (ethereum#18330) * swarm/storage: change Proximity function and add TestProximity test (ethereum#18379) * build: add LGPL license at update-license.go (ethereum#18377) * add LGPL licence at update-licence.go * add empty line * travis, appveyor: bump to Go 1.11.4 (ethereum#18314) * travis, appveyor: bump to Go 1.11.4 * internal/build: revert comment changes * accounts/abi/bind: add optional block number for calls (ethereum#17942) * vendor: vendor/github.com/mattn/go-isatty - add missing files (reported by mksully22) (ethereum#18376) * swarm/docker: Dockerfile for swarm:edge docker image (ethereum#18386) * vendor, crypto, swarm: switch over to upstream sha3 package * vendor: update the entire golang.org/x/crypto dependency * cmd/puppeth: fix panic error when export aleth genesis wo/ precompile-addresses (ethereum#18344) * cmd/puppeth: fix panic error when export aleth genesis wo/ precompile-addresses * cmd/puppeth: don't need to handle duplicate set * eth/tracer: extend create2 (ethereum#18318) * eth/tracer: extend create2 * eth/tracers: fix create2-flaw in prestate_tracer * eth/tracers: fix test * eth/tracers: update assets * accounts/keystore: fix comment typo (ethereum#18395) * A few minor code inspection fixes (ethereum#18393) * swarm/network: fix code inspection problems - typos - redundant import alias * p2p/simulations: fix code inspection problems - typos - unused function parameters - redundant import alias - code style issue: snake case * swarm/network: fix unused method parameters inspections * accounts/keystore: small code simplification (ethereum#18394) * internal/ethapi: ask transaction pool for pending nonce (ethereum#15794) * cmd/geth: support dumpconfig optionally saving to file (ethereum#18327) * Changed dumpConfig function to optionally save to file * Added O_TRUNC flag to file open and cleaned up code * swarm: Shed Index and Uint64Field additions (ethereum#18398) * swarm/storage/mock/test: fix T.Fatal inside a goroutine (ethereum#18399) * swarm, p2p/protocols: Stream accounting (ethereum#18337) * swarm: completed 1st phase of swap accounting * swarm, p2p/protocols: added stream pricing * swarm/network/stream: gofmt simplify stream.go * swarm: fixed review comments * swarm: used snapshots for swap tests * swarm: custom retrieve for swap (less cascaded requests at any one time) * swarm: addressed PR comments * swarm: log output formatting * swarm: removed parallelism in swap tests * swarm: swap tests simplification * swarm: removed swap_test.go * swarm/network/stream: added prefix space for comments * swarm/network/stream: unit test for prices * swarm/network/stream: don't hardcode price * swarm/network/stream: fixed invalid price check * github: remove swarm github codeowners (ethereum#18412) * swarm: Fix T.Fatal inside a goroutine in tests (ethereum#18409) * swarm/storage: fix T.Fatal inside a goroutine * swarm/network/simulation: fix T.Fatal inside a goroutine * swarm/network/stream: fix T.Fatal inside a goroutine * swarm/network/simulation: consistent failures in TestPeerEventsTimeout * swarm/network/simulation: rename sendRunSignal to triggerSimulationRun * swarm/network: remove isproxbin bool from kad.Each* iterfunc (ethereum#18239) * swarm/network, swarm/pss: remove isproxbin bool from kad.Each* iterfunc * swarm/network: restore comment and unskip snapshot sync tests * accounts/abi: tuple support (ethereum#18406) * swarm/network: Rename minproxbinsize, add as member of simulation (ethereum#18408) * swarm/network: Rename minproxbinsize, add as member of simulation * swarm/network: Deactivate WaitTillHealthy, unreliable pending suggestpeer * accounts/abi: Extra slice tests (ethereum#18424) Co-authored-by: weimumu <934657014@qq.com> * p2p/simulations: eliminate concept of pivot (ethereum#18426) * core, eth: fix database version (ethereum#18429) * core, eth: fix database version * eth: polish error message * Stream subscriptions (ethereum#18355) * swarm/network: eachBin now starts at kaddepth for nn * swarm/network: fix Kademlia.EachBin * swarm/network: fix kademlia.EachBin * swarm/network: correct EachBin implementation according to requirements * swarm/network: less addresses simplified tests * swarm: calc kad depth outside loop in EachBin test * swarm/network: removed printResults * swarm/network: cleanup imports * swarm/network: remove kademlia.EachBin; fix RequestSubscriptions and add unit test * swarm/network/stream: address PR comments * swarm/network/stream: package-wide subscriptionFunc * swarm/network/stream: refactor to kad.EachConn * swarm/pot: pot.remove fixed (ethereum#18431) * swarm/pot: refactored pot.remove(), updated comments * swarm/pot: comments updated * swarm/pot: each() functions refactored (ethereum#18452) * accounts/abi: Add tests for reflection ahead of refactor (ethereum#18434) * params: postpone Constantinople due to net SSTORE reentrancy * params, swarm: release Geth v1.8.21 and Swarm v0.3.9 * core, cmd/puppeth: implement constantinople fix, disable EIP-1283 (ethereum#18486) This PR adds a new fork which disables EIP-1283. Internally it's called Petersburg, but the genesis/config field is ConstantinopleFix. The block numbers are: 7280000 for Constantinople on Mainnet 7280000 for ConstantinopleFix on Mainnet 4939394 for ConstantinopleFix on Ropsten 9999999 for ConstantinopleFix on Rinkeby (real number decided later) This PR also defaults to using the same ConstantinopleFix number as whatever Constantinople is set to. That is, it will default to mainnet behaviour if ConstantinopleFix is not set.This means that for private networks which have already transitioned to Constantinople, this PR will break the network unless ConstantinopleFix is explicitly set! * travis, appveyor: bump to Go 1.11.5 (ethereum#18947) * build: tweak debian source package build/upload options (ethereum#18962) dput --passive should make repo pushes from Travis work again. dput --no-upload-log works around an issue I had while uploading locally. debuild -d says that debuild shouldn't check for build dependencies when creating the source package. This option is needed to make builds work in environments where the installed Go version doesn't match the declared dependency in the source package. * p2p/discover, p2p/enode: rework endpoint proof handling, packet logging (ethereum#18963) This change resolves multiple issues around handling of endpoint proofs. The proof is now done separately for each IP and completing the proof requires a matching ping hash. Also remove waitping because it's equivalent to sleep. waitping was slightly more efficient, but that may cause issues with findnode if packets are reordered and the remote end sees findnode before pong. Logging of received packets was hitherto done after handling the packet, which meant that sent replies were logged before the packet that generated them. This change splits up packet handling into 'preverify' and 'handle'. The error from 'preverify' is logged, but 'handle' happens after the message is logged. This fixes the order. Packet logs now contain the node ID. * params: new CHTs (ethereum#18577) * p2p/discover: improve table addition code (ethereum#18974) This change clears up confusion around the two ways in which nodes can be added to the table. When a neighbors packet is received as a reply to findnode, the nodes contained in the reply are added as 'seen' entries if sufficient space is available. When a ping is received and the endpoint verification has taken place, the remote node is added as a 'verified' entry or moved to the front of the bucket if present. This also updates the node's IP address and port if they have changed. * params, swarm/version: Geth 1.8.22-stable, Swarm 0.3.10-stable * eth: make tracers respect pre- EIP 158/161 rule * core: fix error in block iterator (ethereum#18986) * cmd/puppeth: handle pre-set Petersburg number, save changed fork rules * core: repro ethereum#18977 * core: fix pruner panic when importing low-diff-large-sidechain * cmd, core, params: add support for Goerli (cherry picked from commit b0ed083) * build: use SFTP for launchpad uploads (ethereum#19037) * build: use sftp for launchpad uploads * .travis.yml: configure sftp export * build: update CI docs (cherry picked from commit 3de19c8) * common/fdlimit: cap on MacOS file limits, fixes ethereum#18994 (ethereum#19035) * common/fdlimit: cap on MacOS file limits, fixes ethereum#18994 * common/fdlimit: fix Maximum-check to respect OPEN_MAX * common/fdlimit: return error if OPEN_MAX is exceeded in Raise() * common/fdlimit: goimports * common/fdlimit: check value after setting fdlimit * common/fdlimit: make comment a bit more descriptive * cmd/utils: make fdlimit happy path a bit cleaner (cherry picked from commit f48da43) * .travis.yml: fix upload destination (ethereum#19043) (cherry picked from commit edf976e) * build: avoid dput and upload with sftp directly (ethereum#19067) (cherry picked from commit a8ddf7a) * common/fdlimit: fix windows build (ethereum#19068) (cherry picked from commit ba90a4a) * vendor: update syscalls dependency (cherry picked from commit dcc045f) * vendor: pull in upstream syscall fixes for non-linux/arm64 (cherry picked from commit 9d3ea8d) * trie: fix error in node decoding (ethereum#19111) * vendor: update bigcache (cherry picked from commit 37e5a90) * travis.yml: add launchpad SSH public key (ethereum#19115) (cherry picked from commit 75a9314) * build: explicitly force .xz compression (old debuild picks gzip) (ethereum#19118) (cherry picked from commit c0b9c76) * swarm/network: fix data race in TestNetworkID test (ethereum#18460) (cherry picked from commit 96c7c18) * swarm/storage: fix mockNetFetcher data races (ethereum#18462) fixes: ethersphere/swarm#1117 (cherry picked from commit f728837) * cmd/swarm/swarm-snapshot: swarm snapshot generator (ethereum#18453) * cmd/swarm/swarm-snapshot: add binary to create network snapshots * cmd/swarm/swarm-snapshot: refactor and extend tests * p2p/simulations: remove unused triggerChecks func and fix linter * internal/cmdtest: raise the timeout for killing TestCmd * cmd/swarm/swarm-snapshot: add more comments and other minor adjustments * cmd/swarm/swarm-snapshot: remove redundant check in createSnapshot * cmd/swarm/swarm-snapshot: change comment wording * p2p/simulations: revert Simulation.Run from master https://github.com/ethersphere/go-ethereum/pull/1077/files#r247078904 * cmd/swarm/swarm-snapshot: address pr comments * swarm/network/simulations/discovery: removed snapshot write to file * cmd/swarm/swarm-snapshot, swarm/network/simulations: removed redundant connection event check, fixed lint error (cherry picked from commit 34f11e7) * swarm/network: rewrite of peer suggestion engine, fix skipped tests (ethereum#18404) * swarm/network: fix skipped tests related to suggestPeer * swarm/network: rename depth to radius * swarm/network: uncomment assertHealth and improve comments * swarm/network: remove commented code * swarm/network: kademlia suggestPeer algo correction * swarm/network: kademlia suggest peer * simplify suggest Peer code * improve peer suggestion algo * add comments * kademlia testing improvements * assertHealth -> checkHealth (test helper) * testSuggestPeer -> checkSuggestPeer (test helper) * remove testSuggestPeerBug and TestKademliaCase * swarm/network: kademlia suggestPeer cleanup, improved comments * swarm/network: minor comment, discovery test default arg (cherry picked from commit bcb2594) * swarm/network: fix data race warning on TestBzzHandshakeLightNode (ethereum#18459) (cherry picked from commit 81e26d5) * Upload speed (ethereum#18442) (cherry picked from commit 257bfff) * cmd/swarm/swarm-snapshot: disable tests on windows (ethereum#18478) (cherry picked from commit 632135c) * cmd/swarm: use resetting timer to measure fetch time (ethereum#18474) (cherry picked from commit a0b0db6) * cmd/swarm/swarm-smoke: use ResettingTimer instead of Counters for times (ethereum#18479) (cherry picked from commit 5609577) * p2p/simulations: fix data race on swarm/network/simulations (ethereum#18464) (cherry picked from commit 85a79b3) * swarm: bootnode-mode, new bootnodes and no p2p package discovery (ethereum#18498) (cherry picked from commit bbd1203) * swarm: fix a data race on startTime (ethereum#18511) (cherry picked from commit fa34429) * swarm/version: commit version added (ethereum#18510) (cherry picked from commit ad13d2d) * p2p/testing, swarm: remove unused testing.T in protocol tester (ethereum#18500) (cherry picked from commit 2abeb35) * swarm/network: Remove extra random peer, connect test sanity, comments (ethereum#18964) (cherry picked from commit f9401ae) * swarm: fix flaky delivery tests (ethereum#18971) (cherry picked from commit 592bf6a) * swarm/storage: Get all chunk references for a given file (ethereum#19002) (cherry picked from commit 3eff652) * cmd/swarm: hashes command (ethereum#19008) (cherry picked from commit 7f55b0c) * cmd/swarm/swarm-smoke: sliding window test (ethereum#18967) (cherry picked from commit b91bf08) * cmd/swarm/swarm-smoke: remove wrong metrics (ethereum#18970) (cherry picked from commit c5c9cef) * cmd/swarm/swarm-smoke: refactor generateEndpoints (ethereum#19006) (cherry picked from commit d212535) * swarm: Debug API and HasChunks() API endpoint (ethereum#18980) (cherry picked from commit 41597c2) * swarm/storage: fix test timeout with -race by increasing mget timeout (cherry picked from commit 1c3aa8d) * swarm/pss: Remove pss service leak in test (ethereum#18992) (cherry picked from commit 7c60d0a) * swarm/storage/localstore: new localstore package (ethereum#19015) (cherry picked from commit 4f3d22f) * swarm/network: fix data race in fetcher_test.go (ethereum#18469) (cherry picked from commit 19bfcbf) * swarm/network, swarm/storage: Preserve opentracing contexts (ethereum#19022) (cherry picked from commit 0c10d37) * swarm/pss: transition to whisper v6 (ethereum#19023) (cherry picked from commit cde02e0) * swarm: GetPeerSubscriptions RPC (ethereum#18972) (cherry picked from commit 43e1b7b) * swarm/network: refactor simulation tests bootstrap (ethereum#18975) (cherry picked from commit 597597e) * cmd/swarm/global-store: global store cmd (ethereum#19014) (cherry picked from commit 33d0a0e) * swarm: CI race detector test adjustments (ethereum#19017) (cherry picked from commit 27e3f96) * swarm/docker: add global-store and split docker images (ethereum#19038) (cherry picked from commit 6cb7d52) * swarm/pss: mutex lifecycle fixed (ethereum#19045) (cherry picked from commit b30109d) * swarm/storage: fix HashExplore concurrency bug ethersphere#1211 (ethereum#19028) * swarm/storage: fix HashExplore concurrency bug ethersphere#1211 * swarm/storage: lock as value not pointer * swarm/storage: wait for to complete * swarm/storage: fix linter problems * swarm/storage: append to nil slice (cherry picked from commit 3d22a46) * swarm: fix uptime gauge update goroutine leak by introducing cleanup functions (ethereum#19040) (cherry picked from commit d596bea) * swarm: fix network/stream data races (ethereum#19051) * swarm/network/stream: newStreamerTester cleanup only if err is nil * swarm/network/stream: raise newStreamerTester waitForPeers timeout * swarm/network/stream: fix data races in GetPeerSubscriptions * swarm/storage: prevent data race on LDBStore.batchesC ethersphere/swarm#1198 (comment) * swarm/network/stream: fix TestGetSubscriptionsRPC data race ethersphere/swarm#1198 (comment) * swarm/network/stream: correctly use Simulation.Run callback ethersphere/swarm#1198 (comment) * swarm/network: protect addrCountC in Kademlia.AddrCountC function ethersphere/swarm#1198 (comment) * p2p/simulations: fix a deadlock calling getRandomNode with lock ethersphere/swarm#1198 (comment) * swarm/network/stream: terminate disconnect goruotines in tests * swarm/network/stream: reduce memory consumption when testing data races * swarm/network/stream: add watchDisconnections helper function * swarm/network/stream: add concurrent counter for tests * swarm/network/stream: rename race/norace test files and use const * swarm/network/stream: remove watchSim and its panic * swarm/network/stream: pass context in watchDisconnections * swarm/network/stream: add concurrent safe bool for watchDisconnections * swarm/storage: fix LDBStore.batchesC data race by not closing it (cherry picked from commit 3fd6db2) * swarm/storage/netstore: add fetcher cancellation on shutdown (ethereum#19049) swarm/network/stream: remove netstore internal wg swarm/network/stream: run individual tests with t.Run (cherry picked from commit 3ee09ba) * swarm/network: Saturation check for healthy networks (ethereum#19071) * swarm/network: new saturation for implementation * swarm/network: re-added saturation func in Kademlia as it is used elsewhere * swarm/network: saturation with higher MinBinSize * swarm/network: PeersPerBin with depth check * swarm/network: edited tests to pass new saturated check * swarm/network: minor fix saturated check * swarm/network/simulations/discovery: fixed renamed RPC call * swarm/network: renamed to isSaturated and returns bool * swarm/network: early depth check (cherry picked from commit 2af2472) * swarm/storage: fix influxdb gc metrics report (ethereum#19102) (cherry picked from commit 5b8ae78) * swarm/pss: refactoring (ethereum#19110) * swarm/pss: split pss and keystore * swarm/pss: moved whisper to keystore * swarm/pss: goimports fixed (cherry picked from commit 12ca3b1) * p2p, swarm: fix node up races by granular locking (ethereum#18976) * swarm/network: DRY out repeated giga comment I not necessarily agree with the way we wait for event propagation. But I truly disagree with having duplicated giga comments. * p2p/simulations: encapsulate Node.Up field so we avoid data races The Node.Up field was accessed concurrently without "proper" locking. There was a lock on Network and that was used sometimes to access the field. Other times the locking was missed and we had a data race. For example: ethereum#18464 The case above was solved, but there were still intermittent/hard to reproduce races. So let's solve the issue permanently. resolves: ethersphere/swarm#1146 * p2p/simulations: fix unmarshal of simulations.Node Making Node.Up field private in 13292ee broke TestHTTPNetwork and TestHTTPSnapshot. Because the default UnmarshalJSON does not handle unexported fields. Important: The fix is partial and not proper to my taste. But I cut scope as I think the fix may require a change to the current serialization format. New ticket: ethersphere/swarm#1177 * p2p/simulations: Add a sanity test case for Node.Config UnmarshalJSON * p2p/simulations: revert back to defer Unlock() pattern for Network It's a good patten to call `defer Unlock()` right after `Lock()` so (new) error cases won't miss to unlock. Let's get back to that pattern. The patten was abandoned in 85a79b3, while fixing a data race. That data race does not exist anymore, since the Node.Up field got hidden behind its own lock. * p2p/simulations: consistent naming for test providers Node.UnmarshalJSON * p2p/simulations: remove JSON annotation from private fields of Node As unexported fields are not serialized. * p2p/simulations: fix deadlock in Network.GetRandomDownNode() Problem: GetRandomDownNode() locks -> getDownNodeIDs() -> GetNodes() tries to lock -> deadlock On Network type, unexported functions must assume that `net.lock` is already acquired and should not call exported functions which might try to lock again. * p2p/simulations: ensure method conformity for Network Connect* methods were moved to p2p/simulations.Network from swarm/network/simulation. However these new methods did not follow the pattern of Network methods, i.e., all exported method locks the whole Network either for read or write. * p2p/simulations: fix deadlock during network shutdown `TestDiscoveryPersistenceSimulationSimAdapter` often got into deadlock. The execution was stuck on two locks, i.e, `Kademlia.lock` and `p2p/simulations.Network.lock`. Usually the test got stuck once in each 20 executions with high confidence. `Kademlia` was stuck in `Kademlia.EachAddr()` and `Network` in `Network.Stop()`. Solution: in `Network.Stop()` `net.lock` must be released before calling `node.Stop()` as stopping a node (somehow - I did not find the exact code path) causes `Network.InitConn()` to be called from `Kademlia.SuggestPeer()` and that blocks on `net.lock`. Related ticket: ethersphere/swarm#1223 * swarm/state: simplify if statement in DBStore.Put() * p2p/simulations: remove faulty godoc from private function The comment started with the wrong method name. The method is simple and self explanatory. Also, it's private. => Let's just remove the comment. (cherry picked from commit 50b872b) * cmd/swarm/swarm-smoke: Trigger chunk debug on timeout (ethereum#19101) * cmd/swarm/swarm-smoke: first version trigger has-chunks on timeout * cmd/swarm/swarm-smoke: finalize trigger to chunk debug * cmd/swarm/swarm-smoke: fixed httpEndpoint for trigger * cmd/swarm/swarm-smoke: port * cmd/swarm/swarm-smoke: ws not rpc * cmd/swarm/swarm-smoke: added debug output * cmd/swarm/swarm-smoke: addressed PR comments * cmd/swarm/swarm-smoke: renamed track-timeout and track-chunks (cherry picked from commit 62d7688) * contracts/*: golint updates for this or self warning (cherry picked from commit 53b823a) * swarm: Reinstate Pss Protocol add call through swarm service (ethereum#19117) * swarm: Reinstate Pss Protocol add call through swarm service * swarm: Even less self (cherry picked from commit d88c6ce) * swarm/metrics: Send the accounting registry to InfluxDB (ethereum#18470) (cherry picked from commit f28da4f) * cmd/utils: allow for multiple influxdb tags (ethereum#18520) This PR is replacing the metrics.influxdb.host.tag cmd-line flag with metrics.influxdb.tags - a comma-separated key/value tags, that are passed to the InfluxDB reporter, so that we can index measurements with multiple tags, and not just one host tag. This will be useful for Swarm, where we want to index measurements not just with the host tag, but also with bzzkey and git commit version (for long-running deployments). (cherry picked from commit 21acf0b) * params, swarm: release Geth v1.8.23, Swarm v0.3.11 * light, params: update CHTs, integrate CHT for Goerli too * common/fdlimit: fix macos file descriptors for Go 1.12 * all: simplify timestamps to uint64 (ethereum#19372) * all: simplify timestamps to uint64 * tests: update definitions * clef, faucet, mobile: leftover uint64 fixups * ethash: fix tests * graphql: update schema for timestamp * ethash: remove unused variable * trie: there's no point in retrieving the metaroot * travis: update builders to xenial to shadow Go releases * core: re-omit new log event when logs rebirth * core: minor code polishes + rebase fixes * params: set Rinkeby Petersburg fork block (4th May, 2019) * p2p/discover: bump failure counter only if no nodes were provided (ethereum#19362) This resolves a minor issue where neighbors responses containing less than 16 nodes would bump the failure counter, removing the node. One situation where this can happen is a private deployment where the total number of extant nodes is less than 16. Issue found by @Jsying. * eth, les, geth: implement cli-configurable global gas cap for RPC calls (ethereum#19401) * eth, les, geth: implement cli-configurable global gas cap for RPC calls * graphql, ethapi: place gas cap in DoCall * ethapi: reformat log message * params, swarm: release Geth v1.8.24 (noop Swarm 0.3.12) * cmd/geth: fix accidental --rpccorsdomain and --rpcvhosts removal * params, swarm: hotfix Geth v1.8.25 release to restore rpc flags * les: backported new SendTx cost calculation * les: check required message types in cost table * params, swarm: release Geth v1.8.26 (+noop Swarm v0.3.14) * eth, les, light: enforce CHT checkpoints on fast-sync too * params, swarm: release Geth v1.8.27 (noop Swarm v0.3.15) * 1.8.27 with changes redone to compile * fix compile * runner update
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses the issue when loading a snapshot does not wait for all connections to be established.
Changes:
Change 2. is required for change 1. and change 3. is needed because of the change 2.
Change 2. is required as establishing a large number of connections takes exponentially more time on snapshot loading with peer notifications. This change is apparently the correct behaviour in respect to the Discovery option. But ti requires that discovery is enabled in TestWaitTillHealthy test.