feat: expire messages from the cache based on last seen time #513

smrz2001 · 2023-01-06T00:15:22Z

This PR attempts to fix #502.

It also includes a thread-safe copy of the original time cache implementation.

Both the "first seen" (original) and "last seen" (new) implementations have been abstracted behind the TimeCache interface in a new timecache package.

cc @vyzo

blacklist.go

go.mod

pubsub.go

vyzo · 2023-01-10T13:53:18Z

thanks, i was on vacation. I will take a look soon.

timecache/first_seen_cache.go

BigLep · 2023-01-12T16:37:05Z

@vyzo : after this is reviewed/merged, can you cut a new release? (I see this repo isn't part of unified ci: https://github.com/protocol/.github/blob/master/configs/go.json). I'm asking so it can be bubbled up to Kubo for https://github.com/ipfs/kubo/pull/9372/files#diff-a75543978c37023721256291000855b02598e5d72668e203f7372cc4b8c159cfR222

vyzo · 2023-01-12T16:42:09Z

sure

BigLep · 2023-01-13T22:22:04Z

@vyzo : I know you're juggling a lot. Would it be helpful if someone from the EngRes libp2p Stewards team takes a look?

libp2p/go-libp2p-pubsub#513

vyzo · 2023-01-13T23:36:18Z

It has been a little hectic in fvm land recently, but i do have some time to look at this during the weekend.

vyzo · 2023-01-13T23:37:08Z

also, feel free to add steward reviewers, it would help disseminate knowledge.

lidel

@vyzo I eyeballed it, but any chance you take a look at this week?

@smrz2001 quick ask: unify default strategy (right now it is different https://github.com/libp2p/go-libp2p-pubsub/pull/513/files#r1071344615)

vyzo · 2023-01-16T15:20:32Z

yes, yes, my weekend plan didnt materialize, but i have it on the list.

smrz2001 · 2023-01-16T15:36:47Z

@vyzo I eyeballed it, but any chance you take a look at this week?

@smrz2001 quick ask: unify default strategy (right now it is different https://github.com/libp2p/go-libp2p-pubsub/pull/513/files#r1071344615)

Fixed! Thanks, @lidel.

libp2p/go-libp2p-pubsub#513

BigLep · 2023-01-20T22:36:00Z

@MarcoPolo @marten-seemann @Wondertan : any chance you can please help take a look at this? I'd like to get this over the line to get @smrz2001 and team unblocked.

Wondertan · 2023-01-23T09:05:46Z

Happy to review this on the weekends if it will be relevant until then.

vyzo

A couple of issues to decide and a blocker: please dont change the default behaviour!

timecache/first_seen_cache.go

timecache/last_seen_cache.go

vyzo · 2023-01-23T18:01:56Z

Please retain the default behaviour of first seen.

oed · 2023-01-23T19:35:13Z

Please retain the default behaviour of first seen.

@vyzo you said previously (#502 (comment)) that it should be changed. This was the point of the bug that was reported.

vyzo · 2023-01-23T19:49:02Z

Yes, i know, but i had achange of heart regarding the default behaviour.

It is true that it is desirable to use the last seen behaviour for many applications.

However, some of the biggest users of this library are blockchains which can quite easily post reject and deduplicate messages once the timecache period has expired, because of consensus.
That is, after two minutes almost all blockchains can reject older messages reappearing and replays with their consensus based validators (nonce, block height, and so on).

For these applications, changing the default under their feet will not bring any tangible benefit for the increased memory usage.
At the very least i think we should be conservative and let the user choose.

Note that we can also consider making last seen the default a bit further down the line, say in a release or two.
Also, maturity of implementation concerns.

Does this make sense?

oed · 2023-01-23T20:00:00Z

It does make sense. Thanks for explaining your reasoning @vyzo

vyzo

ok, lets merge it and follow up for background gc.

vyzo · 2023-01-23T20:51:17Z

@lidel you are blocking merge.

unblocking

libp2p/go-libp2p-pubsub#513 use last seen time cache implementation

smrz2001 · 2023-01-23T21:36:45Z

ok, lets merge it and follow up for background gc.

Great, thanks so much @vyzo and @lidel!

I opened #515 for the GC follow-up discussion.

* feat: expire messages from the cache based on last seen time * docs: Pubsub.SeenMessagesStrategy Ref. libp2p/go-libp2p-pubsub#513 Co-authored-by: Marcin Rataj <lidel@lidel.org>

* feat: expire messages from the cache based on last seen time (libp2p#513) * feat: expire messages from the cache based on last seen time * chore: minor renaming * fix: messages should not be found after expiration * chore: editorial * fix: use new time cache strategy consistently * fix: default to old time cache and add todo for background gc * chore: update to go-libp2p v0.25 (libp2p#517) * Update to go-libp2p v0.25 * Use go 1.19 * chore: update go version and dependencies (libp2p#516) * fix(timecache): remove panic in first seen cache on Add (libp2p#522) * Refactor timecache implementations (libp2p#523) * reimplement timecache for sane and performant behaviour * remove seenMessagesMx, take advantage of new tc api * fix timecache tests * fix typo * store expiry, don't make life difficult * refactor common background sweep procedure for both impls * add godocs to TimeCache * Default validator support (libp2p#525) * add default validator support * add an implementation for basic seqno as nonce validation * missing return * the nonce belongs to the origin peer * add note about rust predicament * add seqno validator tests * minor test tweak, ensure at least 1ms before replay * Fix Memory Leak In New Timecache Implementations (libp2p#528) * fix bug * add for last seen cache * chore: Update .github/workflows/stale.yml [skip ci] * chore: Update .github/workflows/stale.yml [skip ci] * upgrades libp2p version * upgrades libp2p version --------- Co-authored-by: Mohsin Zaidi <2236875+smrz2001@users.noreply.github.com> Co-authored-by: Marco Munizaga <git@marcopolo.io> Co-authored-by: RichΛrd <info@richardramos.me> Co-authored-by: Hlib Kanunnikov <hlibwondertan@gmail.com> Co-authored-by: vyzo <vyzo@hackzen.org> Co-authored-by: Nishant Das <nishdas93@gmail.com> Co-authored-by: GitHub <noreply@github.com>

* feat: expire messages from the cache based on last seen time (libp2p#513) * feat: expire messages from the cache based on last seen time * chore: minor renaming * fix: messages should not be found after expiration * chore: editorial * fix: use new time cache strategy consistently * fix: default to old time cache and add todo for background gc * chore: update to go-libp2p v0.25 (libp2p#517) * Update to go-libp2p v0.25 * Use go 1.19 * chore: update go version and dependencies (libp2p#516) * fix(timecache): remove panic in first seen cache on Add (libp2p#522) * Refactor timecache implementations (libp2p#523) * reimplement timecache for sane and performant behaviour * remove seenMessagesMx, take advantage of new tc api * fix timecache tests * fix typo * store expiry, don't make life difficult * refactor common background sweep procedure for both impls * add godocs to TimeCache * Default validator support (libp2p#525) * add default validator support * add an implementation for basic seqno as nonce validation * missing return * the nonce belongs to the origin peer * add note about rust predicament * add seqno validator tests * minor test tweak, ensure at least 1ms before replay * Fix Memory Leak In New Timecache Implementations (libp2p#528) * fix bug * add for last seen cache * chore: Update .github/workflows/stale.yml [skip ci] * chore: Update .github/workflows/stale.yml [skip ci] * bump golang.org/x/net from 0.4.0 to 0.7.0 (libp2p#520) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.4.0 to 0.7.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](golang/net@v0.4.0...v0.7.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: topicscore params can't be set for dynamically subscribed topic (libp2p#540) * fix: topicscore params can't be set for a topic subscribed after gossipsub is initialized * chore:address review comments * Revert "fix: topicscore params can't be set for dynamically subscribed topic (libp2p#540)" (libp2p#541) This reverts commit aa5fd79. * remove usage of deprecated peerid.Pretty method (libp2p#542) * chore: update go-libp2p to v0.32 (libp2p#548) * chore: Update .github/workflows/stale.yml [skip ci] * make tidy --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Mohsin Zaidi <2236875+smrz2001@users.noreply.github.com> Co-authored-by: Marco Munizaga <git@marcopolo.io> Co-authored-by: RichΛrd <info@richardramos.me> Co-authored-by: Hlib Kanunnikov <hlibwondertan@gmail.com> Co-authored-by: vyzo <vyzo@hackzen.org> Co-authored-by: Nishant Das <nishdas93@gmail.com> Co-authored-by: GitHub <noreply@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Prem Chaitanya Prathi <chaitanyaprem@gmail.com> Co-authored-by: Sukun <sukunrt@gmail.com>

feat: expire messages from the cache based on last seen time

dfeb78e

smrz2001 mentioned this pull request Jan 6, 2023

seenMessages TTL should be based on last observed time, not first #502

Closed

smrz2001 commented Jan 6, 2023

View reviewed changes

blacklist.go Show resolved Hide resolved

smrz2001 commented Jan 6, 2023

View reviewed changes

go.mod Show resolved Hide resolved

smrz2001 commented Jan 6, 2023

View reviewed changes

pubsub.go Outdated Show resolved Hide resolved

chore: minor renaming

e6faa7b

vyzo self-requested a review January 10, 2023 13:52

smrz2001 added 2 commits January 10, 2023 10:52

fix: messages should not be found after expiration

8571b8a

chore: editorial

04bfcf5

smrz2001 commented Jan 10, 2023

View reviewed changes

timecache/first_seen_cache.go Show resolved Hide resolved

smrz2001 mentioned this pull request Jan 13, 2023

Use updated pubsub time cache implementation ipfs/kubo#9542

Closed

3 tasks

lidel added a commit to smrz2001/kubo that referenced this pull request Jan 13, 2023

chore: go-libp2p-pubsub PR 513

5cd2c3e

libp2p/go-libp2p-pubsub#513

lidel mentioned this pull request Jan 13, 2023

feat: Pubsub.SeenMessagesStrategy ipfs/kubo#9543

Merged

lidel previously requested changes Jan 16, 2023

View reviewed changes

fix: use new time cache strategy consistently

f7c6da6

smrz2001 added a commit to smrz2001/kubo that referenced this pull request Jan 16, 2023

chore: go-libp2p-pubsub PR 513

ebe77ef

libp2p/go-libp2p-pubsub#513

vyzo reviewed Jan 23, 2023

View reviewed changes

timecache/first_seen_cache.go Show resolved Hide resolved

timecache/last_seen_cache.go Show resolved Hide resolved

timecache/last_seen_cache.go Show resolved Hide resolved

fix: default to old time cache and add todo for background gc

c3bfcfa

vyzo approved these changes Jan 23, 2023

View reviewed changes

vyzo requested a review from lidel January 23, 2023 20:50

smrz2001 added a commit to smrz2001/kubo that referenced this pull request Jan 23, 2023

chore: go-libp2p-pubsub PR 513

1fe1734

libp2p/go-libp2p-pubsub#513 use last seen time cache implementation

vyzo merged commit 973fef5 into libp2p:master Jan 24, 2023

smrz2001 deleted the feature/res-849-writing-code-for-libp2p branch January 24, 2023 00:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: expire messages from the cache based on last seen time #513

feat: expire messages from the cache based on last seen time #513

smrz2001 commented Jan 6, 2023 •

edited

Loading

vyzo commented Jan 10, 2023

BigLep commented Jan 12, 2023

vyzo commented Jan 12, 2023

BigLep commented Jan 13, 2023

vyzo commented Jan 13, 2023

vyzo commented Jan 13, 2023

lidel left a comment •

edited

Loading

vyzo commented Jan 16, 2023

smrz2001 commented Jan 16, 2023

BigLep commented Jan 20, 2023

Wondertan commented Jan 23, 2023

vyzo left a comment •

edited

Loading

vyzo commented Jan 23, 2023

oed commented Jan 23, 2023

vyzo commented Jan 23, 2023 •

edited

Loading

oed commented Jan 23, 2023

vyzo left a comment

vyzo commented Jan 23, 2023

smrz2001 commented Jan 23, 2023

feat: expire messages from the cache based on last seen time #513

feat: expire messages from the cache based on last seen time #513

Conversation

smrz2001 commented Jan 6, 2023 • edited Loading

vyzo commented Jan 10, 2023

BigLep commented Jan 12, 2023

vyzo commented Jan 12, 2023

BigLep commented Jan 13, 2023

vyzo commented Jan 13, 2023

vyzo commented Jan 13, 2023

lidel left a comment • edited Loading

Choose a reason for hiding this comment

vyzo commented Jan 16, 2023

smrz2001 commented Jan 16, 2023

BigLep commented Jan 20, 2023

Wondertan commented Jan 23, 2023

vyzo left a comment • edited Loading

Choose a reason for hiding this comment

vyzo commented Jan 23, 2023

oed commented Jan 23, 2023

vyzo commented Jan 23, 2023 • edited Loading

oed commented Jan 23, 2023

vyzo left a comment

Choose a reason for hiding this comment

vyzo commented Jan 23, 2023

smrz2001 commented Jan 23, 2023

smrz2001 commented Jan 6, 2023 •

edited

Loading

lidel left a comment •

edited

Loading

vyzo left a comment •

edited

Loading

vyzo commented Jan 23, 2023 •

edited

Loading