fix(trie): memory leak fix in `lib/trie` #2009

qdm12 · 2021-11-09T16:51:59Z

Changes

Buffer and pool usage fixed and improved
- Fix buffers not put back in pool
- Write to buffer passed as arguments
- Decouple pools for encoding, digests and hashers
Improve sync.Pool usage generally
Improve parallel encoding of branches and remove dependency on sync/x
Do not copy when not needed

Tests

go test -race github.com/ChainSafe/gossamer/lib/trie

Issues

Investigate codebase for memory problems #1973

Primary Reviewer

codecov · 2021-11-09T17:26:03Z

Codecov Report

Merging #2009 (0ae2bf1) into development (8c2993d) will increase coverage by 0.23%.
The diff coverage is 73.98%.

@@               Coverage Diff               @@
##           development    #2009      +/-   ##
===============================================
+ Coverage        60.12%   60.35%   +0.23%     
===============================================
  Files              195      200       +5     
  Lines            26671    27013     +342     
===============================================
+ Hits             16035    16304     +269     
- Misses            8755     8800      +45     
- Partials          1881     1909      +28

Flag	Coverage Δ
unit-tests	`60.35% <73.98%> (+0.23%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
cmd/gossamer/flags.go	`66.66% <ø> (ø)`
cmd/gossamer/main.go	`14.40% <0.00%> (+0.22%)`	⬆️
dot/config.go	`25.89% <21.62%> (-0.74%)`	⬇️
cmd/gossamer/config.go	`74.96% <37.14%> (-2.17%)`	⬇️
internal/pprof/settings.go	`50.00% <50.00%> (ø)`
lib/trie/hash.go	`63.96% <67.93%> (-11.82%)`	⬇️
internal/httpserver/run.go	`83.72% <83.72%> (ø)`
cmd/gossamer/export.go	`88.46% <100.00%> (+0.83%)`	⬆️
cmd/gossamer/utils.go	`74.35% <100.00%> (+0.33%)`	⬆️
dot/node.go	`56.96% <100.00%> (-0.14%)`	⬇️
... and 18 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ce24ea9...0ae2bf1. Read the comment docs.

lib/trie/hash.go

lib/trie/hash_test.go

lib/trie/node.go

noot · 2021-11-18T21:12:39Z

lib/trie/hash.go

+	encodingBuffer.Reset()
+	defer encodingBufferPool.Put(encodingBuffer)
+
+	const parallel = false


is this always false now? can we set it to not false by default or no?

I have no idea. It should work the same, but I left it as is it was for now.

Memory stack wise, I would tend to think that would end up running 100(00)s of goroutines, leading to hundreds of MBs on the stack. I would rather have it run slower for now. We could have it in the future with cpu cores worker goroutines instead so it would be fast and with a low memory footprint, but definitely not in this PR 😉

I'd remove parallel completely then. Not sure why we have this as an option if the caller of this method can never modify it.

I kinda liked the parallel option (I think it increased the sync speed, I did notice it is a bit slower on here, but would need to profile it to really make sure) but yeah either remove it or re-add it as an option I guess

Let's leave it for now, to keep it similar to how it was in this PR. I'll do another PR to refactor it to use N workers with N = cpu cores. That would not use more memory and would be faster.

lib/trie/hash.go

lib/trie/node.go

lib/trie/print.go

Co-authored-by: noot <36753753+noot@users.noreply.github.com>

timwu20 · 2021-11-19T18:36:42Z

lib/trie/hash.go

+var hasherPool = &sync.Pool{
+	New: func() interface{} {
+		hasher, err := blake2b.New256(nil)
+		if err != nil {


I looked at the blake2b code and I don't think this will ever happen if we pass in nil for the key. I'd prefer to omit the error check in this case. Seeing panics in the code gives me a panic. 😅

Well, I agree and disagree because

If someone in the future decides to set a key for whatever future reason, a panic would help out quickly find why it's crashing

blake2b is still in the experimental part of the standard library

Let me know if I'm way too paranoid though 😸

I think linking to this discussion as comment above the panic would be nice.

This is done in #2077 (https://github.com/ChainSafe/gossamer/pull/2077/files#diff-2ab83e082ae5556c9b4a34bd55d1068ffeb45fa6212e848643fcc9f161cf77bbR36)

timwu20 · 2021-11-19T18:46:38Z

lib/trie/hash.go

 // NodeHeader | Extra partial key length | Partial Key | Value
-func (h *hasher) encode(n node) ([]byte, error) {
+func encodeNode(n node, buffer *bytes.Buffer, parallel bool) (err error) {


Suggested change

func encodeNode(n node, buffer *bytes.Buffer, parallel bool) (err error) {

func encodeNodeToBuffer(n node, buffer *bytes.Buffer, parallel bool) (err error) {

In my mind encoding should be an encode() ([]byte, err) receiver function on the node interface. That way you can move all of the actual encoding to the receiver function.

Taking note, will do in another PR. Although:

we need to use *bytes.Buffer to use the sync.Pool (or pass in a byte slice as argument)

we cannot have a method on node. We can have an encode method on branch and leaf, but I would like to do this in another PR once this get merged.

As soon as this gets merged, I'll planning on having packages as

/lib/trie |--- node |--- leaf |--- branch

with exported methods on each implementation.

This is done in #2077

timwu20 · 2021-11-19T18:48:52Z

lib/trie/hash.go

+// encodeLeaf encodes a leaf to the buffer given, with the encoding
+// specified at the top of this package.
+func encodeLeaf(l *leaf, buffer io.Writer) (err error) {
+	l.encodingMu.RLock()


I dislike that this function is modifying the lock of a type here. This logic should be within a receiver function on leaf.

We cannot have a method on leaf to return the encoding l.encoding without copying it, since it could get modified here in encodeLeaf. And I think we should try to reduce copying until we have better memory usage.

encoding := l.getEncoding() if !l.dirty && l.encoding != nil { // here it could be written at the same time as checking the if condition // ...

timwu20 · 2021-11-19T18:50:34Z

lib/trie/hash.go

+		n.encodingMu.Lock()
+		defer n.encodingMu.Unlock()
+
+		n.encoding = make([]byte, buffer.Len())


I don't think we should be setting attributes on the leaf within this function. Could we not call some leaf.encode() method and copy it into the buffer then? Don't we have to setDirty(true) as well?

I don't think we should be setting attributes on the leaf within this function. Could we not call some leaf.encode() method and copy it into the buffer then?

Done in 42f83f1. I think was trying to split out encoding from storing things inside the leaf. Maybe to do in another PR later.

Don't we have to setDirty(true) as well?

I am not sure, it was not there before. It looks like receiver methods on the Trie are the ones using setDirty 🤔 I will definitely look into it while refactoring the trie package in other PRs.

Actually now that I remember, the aim to take this out of encodeLeaf is to expose the copying in order to try to remove it. I prefer to have them higher up in the call stack than at the bottom so I can eventually take them out somehow. For now, this copying defeats the point of the sync.Pool buffers in encodeLeaf for example. I need to investigate more how this all works together first though. I reverted it and added a comment to remove it.

This is placed inside the leaf's Encode() method in #2077 in order to not do type assertion and rely on the node interface.

timwu20 · 2021-11-19T19:20:36Z

lib/trie/hash.go

+	encodingBuffer.Reset()
+	defer encodingBufferPool.Put(encodingBuffer)
+
+	const parallel = false


I'd remove parallel completely then. Not sure why we have this as an option if the caller of this method can never modify it.

lib/trie/hash.go

timwu20 · 2021-11-19T19:27:42Z

lib/trie/node.go

@@ -211,71 +219,94 @@ func (b *branch) setKey(key []byte) {
 	b.key = key
 }

-func (b *branch) encodeAndHash() ([]byte, []byte, error) {
+func (b *branch) encodeAndHash() (encoding, hash []byte, err error) {


why do we have both encodeAndHash functions on the package level and on the node receiver functions?

That's how it was and I wanted to keep deltas not too high so it's still somehow reviewable. Plus we don't have that many 'deep' tests, so I would like to have these before doing more important changes. (tests added in #2049)

I'll take note for another refactor PR though, good point.

This should be addressed to some extent in #2077

noot

nice work! in terms of the memory - it appears after running for ~9h the memory is 10.7GB, so it's improved for sure but still does crawl up, I'm just running the node with ./bin/gossamer --chain polkadot

qdm12 · 2021-11-22T14:10:08Z

@timwu20 our holy CI let all the required workflows pass, so ideally I'll write down further changes to do for other PRs and merge that one, unless it's a critical problem.

timwu20 · 2021-11-22T17:43:03Z

@timwu20 our holy CI let all the required workflows pass, so ideally I'll write down further changes to do for other PRs and merge that one, unless it's a critical problem.

LGTM!

# [0.6.0](v0.5.0...v0.6.0) (2021-12-03) ### Bug Fixes * **babe:** Fix extrinsic format in block. ([#1530](#1530)) ([1a03b2a](1a03b2a)) * **ci:** add missing go-version matrix to fix development branch CI ([#2037](#2037)) ([6babe76](6babe76)) * **cmd/cfg:** Use Babe Lead value from toml config ([#2032](#2032)) ([06aa3e3](06aa3e3)) * cmd/gossamer: Generate random name if --name flag not set ([#1506](#1506)) ([3c05a88](3c05a88)) * confirm block import notifier is closed properly ([#1736](#1736)) ([ad2d85e](ad2d85e)) * **docs:** improve build-spec usage docs ([#1706](#1706)) ([2e164b4](2e164b4)) * **dot/core:** Add only extrinsic during chain reorg. ([#1609](#1609)) ([29413d4](29413d4)) * **dot/core:** Batch process transaction message. ([#1780](#1780)) ([0064836](0064836)) * **dot/core:** check transaction Validity.Propagate field to determine whether to propagate tx ([#1643](#1643)) ([81f23cc](81f23cc)) * **dot/core:** Fix handle transaction message test. ([#1607](#1607)) ([58b8725](58b8725)) * **dot/network, lib/grandpa:** fix handshake decoding and grandpa message handler sigabort ([#1631](#1631)) ([887f72c](887f72c)) * **dot/network, lib/grandpa:** fix node sync, improve devnet finality ([bcc7935](bcc7935)) * **dot/network:** add nil checks in connManager ([#2069](#2069)) ([7f9c042](7f9c042)) * **dot/network:** Check for size when decoding leb128. ([#1634](#1634)) ([d082b9e](d082b9e)) * **dot/network:** check if peer supports protocol ([#1617](#1617)) ([6bf66a4](6bf66a4)) * **dot/network:** decrease DHT find peers interval for gssmr nodes ([#1703](#1703)) ([08516a0](08516a0)) * **dot/network:** fix bugs in notifications protocol handlers; add metrics for inbound/outbound streams ([#2010](#2010)) ([8c2993d](8c2993d)) * **dot/network:** fix dht connection on discovery on devnet ([#2059](#2059)) ([da065b8](da065b8)) * **dot/network:** fix discovery between gossamer nodes ([#1594](#1594)) ([f4c79d3](f4c79d3)) * **dot/network:** fix justification request at head logic ([#1510](#1510)) ([98d1413](98d1413)) * **dot/network:** fix memory allocations with `sizedBufferPool` ([#1963](#1963)) ([e0b126b](e0b126b)) * **dot/network:** Fix missing digest in header ([#2092](#2092)) ([21ea85e](21ea85e)) * **dot/network:** Fix notification handshake and reuse stream. ([#1545](#1545)) ([a632dc4](a632dc4)) * **dot/network:** fix receiving notifications messages from substrate peers ([#1517](#1517)) ([fdf3c53](fdf3c53)) * **dot/network:** fix stream manager tests ([#1683](#1683)) ([e02eca4](e02eca4)) * **dot/network:** implement a handshake timeout ([#1615](#1615)) ([87c2f63](87c2f63)) * **dot/network:** Implement time based handle transaction ([#1942](#1942)) ([dd08424](dd08424)) * **dot/network:** move low reputation peer removal from network ConnManager to peer scoring logic (dot/peerstate) ([#2068](#2068)) ([ac16285](ac16285)), closes [#2039](#2039) * **dot/network:** Return on EOF error while reading stream. ([#1733](#1733)) ([f447eac](f447eac)) * **dot/network:** split stored streams and handshakeData into inbound and outbound ([#1553](#1553)) ([637050b](637050b)) * **dot/network:** update notificationsProtocol handshakeData to sync.Map ([#1492](#1492)) ([22f7269](22f7269)) * **dot/node:** Start websocket server only with `--ws` flag ([#1671](#1671)) ([6ecef3b](6ecef3b)) * **dot/state, lib/babe, lib/trie:** improve syncing between gossamer authority nodes ([#1613](#1613)) ([ca99fbf](ca99fbf)) * **dot/state, lib/grandpa:** update justification and SignedVote handling in database ([#1682](#1682)) ([bbdcd6f](bbdcd6f)) * **dot/state,dot/network:** improve memory usage when syncing ([#1491](#1491)) ([3b2ad8d](3b2ad8d)) * **dot/state:** add StorageState Lock/Unlock API for usage by babe and sync ([#1700](#1700)) ([3c22ace](3c22ace)) * **dot/state:** fix deadlock, fixes bootstrap syncing ([#1959](#1959)) ([dd80c09](dd80c09)) * **dot/state:** fix usage of trie.Snapshot ([#1489](#1489)) ([3880a40](3880a40)) * **dot/state:** track runtime per-block, fix runtime upgrades differing between forks ([#1638](#1638)) ([e133884](e133884)) * **dot/state:** update `*state.BlockState.AddBlockToBlockTree` to store block in `unfinalisedBlocksMap` ([#2006](#2006)) ([55d997f](55d997f)) * **dot/sync:** add nil header checks ([#2099](#2099)) ([a7d4be0](a7d4be0)) * **dot/sync:** fix block request and response logic ([#1907](#1907)) ([9c6283e](9c6283e)) * **dot/sync:** fix creating block response, fixes node sync between gossamer nodes ([#1572](#1572)) ([1328c80](1328c80)) * **dot/telemetry:** refactor telemetry to reduce CPU usage ([#1597](#1597)) ([bc31ac7](bc31ac7)) * **dot/types:** *types.Body to be of type []types.Extrinsic ([#1807](#1807)) ([4c09715](4c09715)) * **dot/types:** fix max value for digest ([#1687](#1687)) ([48405e7](48405e7)) * **dot:** fix `TestNewNode` ([#2070](#2070)) ([42908d0](42908d0)) * fix edit link ([#1507](#1507)) ([5089327](5089327)) * fix Kusama sync; add storageState lock in core.HandleTransactionMessage ([#1783](#1783)) ([1d688e4](1d688e4)) * **lib/babe, lib/runtime/wasmer:** fixes for v0.9.8+ runtime ([#2075](#2075)) ([2f9f80c](2f9f80c)) * **lib/babe:** add `--babe-lead` flag, update epoch handling logic ([#1895](#1895)) ([7abcce6](7abcce6)) * **lib/babe:** add pre-runtime digest before calling initialize_block ([#1581](#1581)) ([c1b26d3](c1b26d3)) * **lib/babe:** always use 2/3 of slot to produce block, re-add potentially valid txs to queue ([#1679](#1679)) ([cf93ad3](cf93ad3)) * **lib/babe:** call AddBlock in BABE synchronously ([#1585](#1585)) ([86acc43](86acc43)) * **lib/babe:** fix BABE state storing after building block ([#1536](#1536)) ([1a3dea2](1a3dea2)) * **lib/babe:** fix err log ([#1801](#1801)) ([a96f06a](a96f06a)) * **lib/babe:** fix setting first slot of network, fix loading BABE epoch params ([#1640](#1640)) ([5c3dbfe](5c3dbfe)) * **lib/babe:** fix timing for transition between epochs ([#1636](#1636)) ([57027db](57027db)) * **lib/blocktree:** fix blocktree bug ([#2060](#2060)) ([c17b53a](c17b53a)) * **lib/blocktree:** fix potential nil pointer dereference in `HighestCommonAncestor`, core `handleBlocksAsync` ([#1993](#1993)) ([f7f4463](f7f4463)) * **lib/blocktree:** fix setting leaves after blocktree pruning ([#1605](#1605)) ([58c0854](58c0854)) * **lib/blocktree:** removes the inconsistency to choose a deepest leaf ([#2094](#2094)) ([43d68e3](43d68e3)) * **lib/crypto/ed25519:** update ed25519 to use go-schnorrkel bip39 derivation ([#1488](#1488)) ([dfb95d2](dfb95d2)) * **lib/genesis:** Update missing and incorrect fields in genesis file. ([#1681](#1681)) ([8207704](8207704)) * **lib/grandpa:** fix grandpa stall and various bugs ([#1708](#1708)) ([67c93f4](67c93f4)) * **lib/grandpa:** fix grandpa vote message switch ([#2095](#2095)) ([461890c](461890c)) * **lib/grandpa:** fix threshold checking to be strictly greater than 2/3 ([#1891](#1891)) ([66ffe51](66ffe51)) * **lib/grandpa:** use `defaultGrandpaInterval` if not set, fixes error on startup ([#1982](#1982)) ([75627b5](75627b5)) * **lib/runtime/life:** remove import C from life ([#1923](#1923)) ([ed507d2](ed507d2)) * **lib/runtime:** update HOST_API_TEST_RUNTIME_URL to reference specific commit ([#1885](#1885)) ([666ed06](666ed06)) * **log-levels:** do not ignore configuration file log levels ([#2016](#2016)) ([80879b2](80879b2)) * pending bubble hidden after block included ([#1592](#1592)) ([5826322](5826322)) * persist node name ([#1543](#1543)) ([88b88f2](88b88f2)) * **pprof:** only run pprof service if enabled ([#2073](#2073)) ([55669c5](55669c5)) * **release:** Trigger release when pushed to main branch. ([#1566](#1566)) ([d445c97](d445c97)) * **rpc/subscription:** subscribe runtime version notify when version changes ([#1686](#1686)) ([9a76d39](9a76d39)) * Staging CI workflow ([#2034](#2034)) ([84ec792](84ec792)) * **trie:** memory leak fix in `lib/trie` ([#2009](#2009)) ([0ad5eb7](0ad5eb7)) * update deprecated package ([#1603](#1603)) ([f195204](f195204)) * update go-schnorrkel version ([#1557](#1557)) ([b86c7ff](b86c7ff)) * update gssmr genesis to use pallet_babe::SameAuthoritiesForever ([#1696](#1696)) ([fb0a751](fb0a751)) * update HOST_API_TEST_RUNTIME_URL ([#1898](#1898)) ([2ef59a8](2ef59a8)) * **utils:** create a specific folder for database ([#1598](#1598)) ([8c67795](8c67795)) ### Features * add --chain dev option ([#1561](#1561)) ([04a2969](04a2969)) * Add properties and chainId on build-spec command ([#1520](#1520)) ([b18290c](b18290c)) * cmd: implement import-runtime subcommand ([#1483](#1483)) ([d82b2da](d82b2da)) * **cmd/gossamer:** implement --telemetry-url parameter ([#1890](#1890)) ([b202e89](b202e89)), closes [#1502](#1502) * **cmd:** implement offline pruning of state trie ([#1564](#1564)) ([af9c925](af9c925)) * **devnet:** Local Gossamer Devnet ([#2008](#2008)) ([a520001](a520001)) * **dot/network, lib/grandpa:** request justification on receiving NeighbourMessage, verify justification on receipt ([#1529](#1529)) ([e1f9f42](e1f9f42)) * **dot/network:** Add cache for network message. ([#1511](#1511)) ([accaf69](accaf69)) * **dot/network:** add propagate return bool to messageHandler func type to determine whether to propagate message or not ([#1555](#1555)) ([0d6f488](0d6f488)) * **dot/network:** implement persistent peers functionality ([#1512](#1512)) ([7850532](7850532)) * **dot/network:** implement streamManager to cleanup not recently used streams ([#1611](#1611)) ([ba861bf](ba861bf)) * **dot/network:** request block justifications when near head ([#1499](#1499)) ([ae7012b](ae7012b)) * **dot/peerset:** Implement peer scoring ([#1791](#1791)) ([1c989ad](1c989ad)) * **dot/rpc/modules:** add `system_addReservedPeer` and `system_removeReservedPeer` RPC call ([#1712](#1712)) ([dba5922](dba5922)) * **dot/rpc:** Add `system_localListenAddresses` RPC call ([#1689](#1689)) ([c981d2e](c981d2e)) * **dot/rpc:** Implement `childstate_getKeys` rpc call ([#1800](#1800)) ([9b2f41e](9b2f41e)) * **dot/rpc:** implement sync_state_genSyncSpec RPC call ([#1827](#1827)) ([2186caf](2186caf)) * **dot/state:** implement online pruning of historical state tries ([#1596](#1596)) ([3eb9399](3eb9399)) * **dot/sync:** implement codeSubstitutes ([#1635](#1635)) ([d87aaeb](d87aaeb)) * dot/telemetry: Implement basic telemetry connection ([#1497](#1497)) ([fcb4159](fcb4159)) * **dot/telemetry:** Added connection retry ([#1904](#1904)) ([579a791](579a791)) * **dot/telemetry:** Added more telemetry messages in grandpa client ([#2043](#2043)) ([2e57d15](2e57d15)), closes [#1841](#1841) [#1842](#1842) * **dot/telemetry:** implement notify.finalized telemetry interface ([#1877](#1877)) ([de1a60d](de1a60d)) * **dot/telemetry:** implement substrate_number_leaves metrics ([#1926](#1926)) ([69823c0](69823c0)) * **dot/telemetry:** implement telemetry message network_state ([#1618](#1618)) ([a81844e](a81844e)) * **flags:** read log levels from flags ([#1953](#1953)) ([9694e46](9694e46)) * implement ext_default_child_storage_storage_kill_version_2 ([#1799](#1799)) ([c2908ae](c2908ae)) * implement ext_offchain_index_set_version_1 for wasmer runtime ([#1739](#1739)) ([96c30a6](96c30a6)) * **lib/babe:** add check of types.ConfigData.SecondarySlots for disabling secondary verification ([#1910](#1910)) ([cd27ae4](cd27ae4)) * **lib/grandpa:** fully verify justifications using GrandpaState ([#1544](#1544)) ([028d25e](028d25e)) * **lib/grandpa:** Include equivocatory nodes while creating justification ([#1911](#1911)) ([aca86b6](aca86b6)) * **lib/grandpa:** send NeighbourMessage to peers ([#1558](#1558)) ([322ccf9](322ccf9)) * **lib/runtime/wasmer:** implement ext_default_child_storage_storage_kill_version_3 ([#1878](#1878)) ([a719a60](a719a60)) * **lib/runtime/wasmer:** implement ext_offchain_local_storage_version_1 ([#1821](#1821)) ([0f63b17](0f63b17)) * **lib/runtime:** Implement `ext_offchain_http_request_add_header_version_1` host function ([#1994](#1994)) ([0a30b3d](0a30b3d)) * **lib/runtime:** Implement `ext_offchain_http_request_start_version_1` host function ([#1947](#1947)) ([974b1fc](974b1fc)) * **lib/runtime:** Implement `trie_blake2_256_verify_proof` host function ([#1920](#1920)) ([506565d](506565d)) * **lib/trie:** Implement `verify_proof` function ([#1883](#1883)) ([67bb5ef](67bb5ef)) * **lib/trie:** Implement limit for trie.ClearPrefix ([#1905](#1905)) ([becec9e](becec9e)) * **lib/trie:** Parallel hash trie. ([#1657](#1657)) ([22827e7](22827e7)) * **pprof:** Pprof HTTP server service ([#1991](#1991)) ([ce24ea9](ce24ea9)) * **rpc/subscription:** implement state_unsubscribeStorage ([#1574](#1574)) ([7574f10](7574f10)) * **rpc:** Implement `childstate_getChildStorage` RPC call ([#1832](#1832)) ([3d949f2](3d949f2)) * **rpc:** Implement `childstate_getStorageHash` RPC call ([#1805](#1805)) ([e539bd3](e539bd3)) * **rpc:** Implement `childstate_getStorageSize` RPC call ([#1810](#1810)) ([a04deb6](a04deb6)) * **rpc:** Implement `payment_queryInfo` RPC call ([#1826](#1826)) ([7a5deec](7a5deec)) * **rpc:** Implement `state_getReadProof` rpc call ([#1768](#1768)) ([865f80f](865f80f)) * **runtime:** implement custom logging handler that print function name ([#1825](#1825)) ([2b1276d](2b1276d)) * **telemetry:** send telemetry messages when GRANDPA receieves commit or vote messages ([#2015](#2015)) ([7bf40e1](7bf40e1)), closes [#1840](#1840) [#1839](#1839) [#1838](#1838) * **telemetry:** send txpool.import telemetry msg ([#1966](#1966)) ([ffc81bf](ffc81bf)) ### Reverts * Revert "feat(dot/rpc) implement `author_hasSessionKeys` RPC call (#1704)" (#1714) ([65380fd](65380fd)), closes [#1704](#1704) [#1714](#1714)

github-actions · 2021-12-03T21:04:38Z

🎉 This PR is included in version 0.6.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

- Buffer and pool usage fixed and improved - Fix buffers not put back in pool - Write to buffer passed as arguments - Decouple pools for encoding, digests and hashers - Improve `sync.Pool` usage generally - Improve parallel encoding of branches and remove dependency on `sync/x` - Do not copy when not needed

I am Quentin Mc Gaw, a software engineer working the Go Polkadot host **Gossamer**. I have been working full time on Gossamer since October 2021, mostly on the state trie and storage. I have also made a [few minor pull requests](https://github.com/w3f/polkadot-spec/pulls?q=is%3Apr+is%3Aclosed+author%3Aqdm12) to the Polkadot specification repository. I am requesting to join the Fellowship at rank 1. ## Main contributions ### Gossamer - Fix memory leaks - Trie encoding buffer pools usage fixed [#2009](ChainSafe/gossamer#2009) - Fix state map of tries memory leak [#2286](ChainSafe/gossamer#2286) - Fix sync benchmark [#2234](ChainSafe/gossamer#2234) - Trie proof fixes ([#2604](ChainSafe/gossamer#2604), [#2661](ChainSafe/gossamer#2661)) - Fix end to end tests orchestration ([#2470](ChainSafe/gossamer#2470), [#2452](ChainSafe/gossamer#2452), [#2385](ChainSafe/gossamer#2385), [#2370](ChainSafe/gossamer#2370)) - State trie statistics ([#2378](ChainSafe/gossamer#2378), [#2310](ChainSafe/gossamer#2310), [#2272](ChainSafe/gossamer#2272)) - State trie fixes and improvements - Only deep copy nodes when mutation is certain [#2352](ChainSafe/gossamer#2352) and [#2223](ChainSafe/gossamer#2223) - Only deep copy necessary fields of a node [#2384](ChainSafe/gossamer#2384) - Use Merkle values for database keys instead of always hash [#2725](ChainSafe/gossamer#2725) - Opportunistic parallel Merkle value commputing [#2081](ChainSafe/gossamer#2081) - Grandpa capped number of tracked messages ([#2490](ChainSafe/gossamer#2490), [#2485](ChainSafe/gossamer#2485)) - Add pprof HTTP service for profiling [#1991](ChainSafe/gossamer#1991) Ongoing work: - State trie lazy loading and caching - State trie v1 support ([#2736](ChainSafe/gossamer#2736), [#2747](ChainSafe/gossamer#2747), [#2687](ChainSafe/gossamer#2687), [#2686](ChainSafe/gossamer#2686), [#2685](ChainSafe/gossamer#2685), [#2673](ChainSafe/gossamer#2673), [#2611](ChainSafe/gossamer#2611), [#2530](ChainSafe/gossamer#2530)) ### Polkadot specification ➡️ [Pull requests from qdm12](https://github.com/w3f/polkadot-spec/pulls?q=is%3Apr+is%3Aclosed+author%3Aqdm12)

qdm12 changed the title ~~Qdm12 mem hasher fix~~ fix(trie): memory leak fix in lib/trie Nov 9, 2021

qdm12 force-pushed the qdm12-mem-hasher-fix branch from 1c5042a to bee1fac Compare November 12, 2021 13:33

qdm12 mentioned this pull request Nov 16, 2021

testing(lib/trie): change lib/trie slow and intensive benchmark-oriented tests to benchmarks #2031

Merged

qdm12 force-pushed the qdm12-mem-hasher-fix branch from b26cb37 to 0bc4a88 Compare November 16, 2021 21:47

qdm12 marked this pull request as ready for review November 16, 2021 22:20

qdm12 requested review from arijitAD, EclesioMeloJunior, edwardmack, jimjbrettj, kishansagathiya, noot, omar391 and timwu20 as code owners November 16, 2021 22:20

qdm12 mentioned this pull request Nov 17, 2021

Profile memory usage and describe plan of action to address issues #1936

Closed

EclesioMeloJunior requested changes Nov 17, 2021

View reviewed changes

lib/trie/hash.go Show resolved Hide resolved

lib/trie/hash.go Outdated Show resolved Hide resolved

lib/trie/hash_test.go Show resolved Hide resolved

lib/trie/hash_test.go Show resolved Hide resolved

lib/trie/node.go Show resolved Hide resolved

qdm12 force-pushed the qdm12-mem-hasher-fix branch 3 times, most recently from 03a0935 to 6673e86 Compare November 18, 2021 14:51

EclesioMeloJunior approved these changes Nov 18, 2021

View reviewed changes