Added subscription specific configurations to PubSub #171

aschmahmann · 2019-04-08T14:39:32Z

There's still work to be done here, both in documentation and some design work. It'd be great to get some feedback on the direction this is going though.

Overall the additions are:

Added the ability to have subscription based options
Added some configurable parameters to gossipsub
Added a gossipsub last-writer-wins-style persistence configuration

Added simple configurability to gossipsub Added a gossipsub last-writer-wins persistence configuration

raulk · 2019-04-08T14:52:21Z

Hey @aschmahmann, thanks for the contribution! 🎉 Could you link to an issue where the need and rationale behind this was discussed? Thanks!

vyzo

Some very passing comments from first reading.
What's the rationale for these changes?

gossipsub.go

vyzo · 2019-04-08T15:28:55Z

gossipsub.go

 	for _, topic := range msg.GetTopicIDs() {
+		config, ok := gs.topicConfigs[topic]


that's just backwards; can't have a config object handling hte most important function of the router!

Isn't that the whole premise of every inversion of control system? We could create new routers and pubsub instances for every configuration option instead of using one pubsub instance.

However, that means

having multiple heartbeat tickers running at the same time

not reusing any of the gossipsub structures across implementations

ending up with weird multiple tenancy issues unless we change PubSub more. For example, if right now you create a FloodSub and GossipSub instance on the same node you end up with overly excessive message passing because GossipSub is backwards compatible with FloodSub and will handle both messages. Either we need one router to handle multiple configurations or we need to make PubSub able to handle backwards compatible protocols, this way involves less protocol work.

gossipsubimpl.go

vyzo · 2019-04-08T15:30:10Z

subscription.go

-	cancelCh chan<- *Subscription
-	err      error
+	topic         string
+	topicProtocol protocol.ID


I don't think this belongs here.

I disagree, there is in PubSub a declaration type SubOpt func(sub *Subscription) error that is currently almost unusable because Subscription has no fields to work with.

I assume that pattern taken with both Option (for PubSub) and SubOpt (for Subscriptions) is designed to allow adding fields to the struct, and functionality as option wrappers, without breaking the struct contract/interface. Is that incorrect, if so how is SubOpt supposed to be used and where would you put the relevant topic/Subscription information?

aschmahmann

The main rationale for having a persistent pubsub is at #42. I also briefly give some reasons for needing this at #161.

gossipsub.go

gossipsub_test.go

gossipsubimpl.go

pubsub.go

aschmahmann · 2019-04-08T15:20:03Z

gossipsub.go

-		for p := range tmap {
-			if gs.peers[p] == FloodSubID {
-				tosend[p] = struct{}{}
+		if withFloodSubPeers{


The various configurations that I've implemented so far could likely all be done in floodsub, episub, etc. Two caveats are 1) we'd need to allow the layer above the router implementations to have more access to the internals (e.g. which peers I'm syncing/sending messages to for a given topic, some control over limiting or manually triggering a broadcast) 2) The existing less performant implementations (e.g. floodsub) would do extremely poorly for some of these configuration options (e.g. persistence) anyway.

Additionally, in future implementations of say episub the configs used by gossipsub should be reusable.

Given that we're ok breaking the router implementations/interfaces a bit do you think I should try and setup some of the configurations (e.g. persistence) to work across all router implementations or is leaving the configurations as something to be plugged into future routers (e.g. episub) sufficient?

that's fine, although I don't know how useful it will be for floodsub.

aschmahmann · 2019-04-08T15:22:36Z

gossipconfig.go

+	Put(msg *pb.Message)
+}
+
+type ClassicGossipSubConfiguration struct {


Up for a better name if you've got one

vyzo · 2019-04-08T15:48:04Z

Just a note: you are changing very carefully thought-out code, don't go doing stuff willy-nilly!

aschmahmann · 2019-04-08T16:40:56Z

Just a note: you are changing very carefully thought-out code, don't go doing stuff willy-nilly!

Nothing here has been done "willy-nilly". Everything I've done so far has been targeted at making sure "user space" around PubSub and gossipsub has been minimally affected (which is true for anyone using the GossipSub, FloodSub, ... constructors), while at the same time allowing for changes that are needed by consumers of PubSub (e.g. IPNS).

I see from some of your comments that we're actually ok with some breaking changes within the PubSub system which should certainly help clean up the code.

I respect that a lot of thought went into the PubSub system and is part of why I'd like to enable users to tinker with it from user space as much as possible so they don't end up trying to copy-paste or fork pubsub. I think it'd be great to have some document discussing how we'd like to enable people to extend PubSub and also consider whether the configuration options I'm putting in are sufficient or just sufficient for the cases that I'm currently considering.

raulk · 2019-04-09T11:08:19Z

@aschmahmann could you open an RFC discussion in libp2p/specs? Please outline context, motivation, rationale, goals, etc. Then link to this PR as a sample implementation. Inferring all that from various issues is less than ideal and is subject to bias and interpretation. Also, not all libp2p implementers watch all go repos, so we elevate protocol discussions to the specs repo. libp2p is a specs-driven project so any changes introduced here would have to be preambled by changes in specs.

aschmahmann · 2019-04-09T14:03:06Z

@raulk There are 3 potential specs I could add, and would like some advice on which ones to go with. There's: 1) PubSub patch 2) GossipSub patch 3) Something about Last-Writer-Wins persistence

There doesn't appear to be anywhere in the PubSub spec to add anything interesting from my work, although I'm up for any suggestions. I haven't touched any wire protocols and there's no mention in the spec on the Join function or on any of the options in the go implementation (e.g. Blacklist).
I can patch the GossipSub spec to allow for Joining a topic with a particular protocol ID and forwarding certain functions to the particular configurations. However, I need confirmation from @vyzo that the inversion of control mechanism I'd like to use is ok. While IoC is an implementation detail it affects whether the change occurs at the PubSub or GossipSub layers. Given that there's nowhere in the PubSub spec for this to go it seems to imply that either the PubSub spec is missing info or the GossipSub spec has too much implementation specific info.
Happy to write this up, although I need to do a little thinking first because since @vyzo suggested it was possible for me to break the PubSubRouter interfaces it might be useful to implement LWW at the PubSub level instead of the GossipSub level. Note that if it's added at the PubSub level it's unclear where to put the spec changes, as described above.

raulk · 2019-04-09T21:38:12Z

@aschmahmann I would start with an issue in libp2p/specs theorising context (what is the current situation), description of change (what you want to change), motivation/rationale (why you want to change it), proposed design (how you want to change it), proposed next steps.

Then the community can gauge if this is something of interest across implementations, and can advice how to best integrate it.

aschmahmann · 2019-04-15T14:56:45Z

@raulk @vyzo, I published a high level overview of why and what I want to change at the issue above (#175). As you'll notice there's limited spec change proposals (although if my LWW implementation is going to be part of libp2p it may want to get its own spec proposal at some point).

…d be re-emitted to (i.e. not initial message propagation, but rebroadcasting messages) Moved configuration changes to gossipsub into relevant interfaces instead of Options (makes code much cleaner at the expense of breaking the interfaces)

aschmahmann · 2019-04-23T16:32:58Z

@vyzo while PubSub's Subscribe function takes in Options (and as a result can be setup for configuration) the Publish function does not. I was thinking of adding a PublishWithOptions function to accommodate this without breaking anyone who is using a default Publish currently. What do you think?

vyzo · 2019-04-23T16:35:07Z

what options do you want to pass to Publish?

aschmahmann · 2019-04-23T16:49:43Z

For now likely just WithProtocol(proto protocol.ID) as I am with Subscribe

vyzo · 2019-04-23T16:55:38Z

that won't work, the protocol ID is a property of the stream.

aschmahmann · 2019-04-23T17:39:11Z

Yes, it is a property of the stream. In Master currently the way that you would setup a routing system with a different configuration (e.g. adding persistence) is by creating a new PubSubRouter with a new protocol ID.

The implementation here is similar except that instead of having one PubSub instance per router I am effectively putting multiple routers into a single PubSub instance and associating each topic with a particular router. If you look at what I'm doing with Subscribe and Join it's effectively this. However, because you can Publish to a topic without first subscribing I should be able to set the router that a topic is using from the Publish (or PublishWithOptions) function as well.

aschmahmann · 2019-04-23T17:46:18Z

gossipsub.go

-		if !ok || ok {
-			gs.pushGossip(p, &pb.ControlIHave{TopicID: &topic, MessageIDs: mids})
-		}
+	emitPeers := config.GetEmitPeers(gs.wrapGetPeers(topic), peers)


@vyzo adding an EmittingStrategy to the configuration was one way to allow developers to choose which peers messages will get rebroadcasted to.

vyzo · 2019-04-24T08:28:48Z

I am effectively putting multiple routers into a single PubSub instance and associating each topic with a particular router.

You are killing gossip piggybacking with this most likely.

vyzo · 2019-04-24T09:57:10Z

Also, can we call Config something else? It's not an apt name when it encompasses all the fundamental actions of the protocol. Perhaps call it Strategy?

aschmahmann · 2019-04-24T14:30:36Z

You are killing gossip piggybacking with this most likely.

I'm actually think I'm doing the opposite. If we had to make a new PubSub instance for every configuration then we'd be stuck with multiple heartbeats each sending out there own piggybacked data which means that if A and B were connected by two routers then we'd end up with two messages instead of one message with both the data from A and B piggybacked onto it.

aschmahmann · 2019-04-24T14:33:23Z

Also, can we call Config something else? It's not an apt name when it encompasses all the fundamental actions of the protocol. Perhaps call it Strategy?

I don't think it encompasses all the fundamental actions of the protocol (I'd argue that graft + prune is the main component of gossipsub). However, as I don't have strong feelings about the naming here, I'm fine with Strategy or something more specific like XStrategy where X is a little more informative (especially since I've introduced an EmitterStrategy already).

aschmahmann · 2019-04-24T14:58:58Z

gossipsub.go

+}
+
+func (gs *GossipSubRouter) AddGossipPeers(tosend map[peer.ID]struct{}, topics []string, withFloodSubPeers bool) {
+	for _, topic := range topics {
 		// any peers in the topic?
 		tmap, ok := gs.p.topics[topic]
 		if !ok {
 			continue


@vyzo is it intentional that during a Publish we do not initialize gs.fanout[topic] if there are no peers in the topic (and we're not subscribed/there's no gs.mesh[topic])? By not doing this we ensure that we will not rebroadcast (via emitGossip) our published message even once we come across new peers. This is less relevant in the current implementation since the republishing window is relatively short (5 heartbeat intervals which is 500ms), however it might still be worth correcting this even for our existing implementation.

They won't be added to fanout unless there is a new message published.
Also, the heartbeat is 1 second, so 5 intervals are 5s.

My bad on the interval timing, was looking at heartbeat initial delay by accident. Ok, so the following sequence of events in the current implementation is intentional?

Scenario 1:

A publishes M to GossipSub topic T, but has no peers

B subscribes to T

B connects to A

Nothing happens, B never receives M (even if within the cache history period)

Scenario 2:

A publishes M to GossipSub topic T, but has no peers

B1...B20 subscribes to T (important that it's more than GossipSubD=6, as mentioned above)

B1...B20 connects to A

Some of the B's receive M and some do not

Scenario 3:

A publishes M to GossipSub topic T, but has no peers

Wait 6 seconds

B1...B20 subscribes to T

B1...B20 connects to A

Nothing happens, none of the B's receives M

Yes, that's fine.

That psudo-use of a history buffer seems pretty suspect to me, but that's fine I can preserve that behavior and just modify the persistence version.

To clarify, I don't really see what's gained by having B not receive M in scenario 1.

It's not designed to be a reliability mechanism; it's designed to help the network connect when the mesh is not fully connected for one reason or the other.

Renamed configurations to strategies Added OnGraft function to GossipStrategies LWW strategy now requests the latest topic data during OnGraft

vyzo · 2020-03-23T10:39:13Z

Closing as this is stale.

Added subscription specific configurations to PubSub

3f32612

Added simple configurability to gossipsub Added a gossipsub last-writer-wins persistence configuration

vyzo reviewed Apr 8, 2019

View reviewed changes

aschmahmann mentioned this pull request Apr 8, 2019

Reliable and/or Persistent Pubsub #42

Closed

aschmahmann commented Apr 8, 2019

View reviewed changes

aschmahmann mentioned this pull request Apr 15, 2019

PubSub & GossipSub configuration proposal #175

Open

aschmahmann commented Apr 23, 2019

View reviewed changes

aschmahmann commented Apr 24, 2019

View reviewed changes

This was referenced May 1, 2019

IPNS over PubSub Pinning ipfs-shipyard/integration-mini-projects#4

Open

IPNS Benchmarking ipfs-shipyard/integration-mini-projects#5

Open

Simplified LWW strategy by utilizing go-libp2p-record validators.

de12811

Renamed configurations to strategies Added OnGraft function to GossipStrategies LWW strategy now requests the latest topic data during OnGraft

aschmahmann mentioned this pull request May 3, 2019

Subscribe to PubSub topic before Publishing libp2p/go-libp2p-pubsub-router#30

Merged

marcinczenko mentioned this pull request May 24, 2019

IPNS names do not resolve when published with different key than self ipfs/kubo#6360

Closed

aschmahmann mentioned this pull request Jun 4, 2019

Add Discovery #184

Merged

This was referenced Jun 7, 2019

Add the ability to handle newly subscribed peers #190

Merged

IPNS over PubSub as an independent transport ipfs/kubo#6447

Closed

aschmahmann mentioned this pull request Jul 3, 2019

Add Persistence Layer on top of PubSub libp2p/go-libp2p-pubsub-router#33

Merged

aschmahmann mentioned this pull request Aug 5, 2019

Persistence with PubSub libp2p/go-libp2p#698

Closed

aschmahmann mentioned this pull request Oct 2, 2019

Refactor to allow out-of-tree router implementations #199

Closed

vyzo closed this Mar 23, 2020

libp2p deleted a comment from bitcard Feb 28, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added subscription specific configurations to PubSub #171

Added subscription specific configurations to PubSub #171

aschmahmann commented Apr 8, 2019

raulk commented Apr 8, 2019

vyzo left a comment

vyzo Apr 8, 2019

aschmahmann Apr 8, 2019

vyzo Apr 8, 2019

aschmahmann Apr 8, 2019

aschmahmann left a comment

aschmahmann Apr 8, 2019

aschmahmann Apr 8, 2019

vyzo Apr 8, 2019

aschmahmann Apr 8, 2019

vyzo commented Apr 8, 2019

aschmahmann commented Apr 8, 2019

raulk commented Apr 9, 2019

aschmahmann commented Apr 9, 2019

raulk commented Apr 9, 2019

aschmahmann commented Apr 15, 2019

aschmahmann commented Apr 23, 2019

vyzo commented Apr 23, 2019

aschmahmann commented Apr 23, 2019

vyzo commented Apr 23, 2019

aschmahmann commented Apr 23, 2019

aschmahmann Apr 23, 2019

vyzo commented Apr 24, 2019

vyzo commented Apr 24, 2019

aschmahmann commented Apr 24, 2019

aschmahmann commented Apr 24, 2019 •

edited

Loading

aschmahmann Apr 24, 2019 •

edited

Loading

vyzo Apr 24, 2019

aschmahmann Apr 24, 2019 •

edited

Loading

vyzo Apr 24, 2019

aschmahmann Apr 24, 2019 •

edited

Loading

vyzo Apr 25, 2019

vyzo commented Mar 23, 2020

		for _, topic := range msg.GetTopicIDs() {
		config, ok := gs.topicConfigs[topic]

Added subscription specific configurations to PubSub #171

Added subscription specific configurations to PubSub #171

Conversation

aschmahmann commented Apr 8, 2019

raulk commented Apr 8, 2019

vyzo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aschmahmann left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vyzo commented Apr 8, 2019

aschmahmann commented Apr 8, 2019

raulk commented Apr 9, 2019

aschmahmann commented Apr 9, 2019

raulk commented Apr 9, 2019

aschmahmann commented Apr 15, 2019

aschmahmann commented Apr 23, 2019

vyzo commented Apr 23, 2019

aschmahmann commented Apr 23, 2019

vyzo commented Apr 23, 2019

aschmahmann commented Apr 23, 2019

Choose a reason for hiding this comment

vyzo commented Apr 24, 2019

vyzo commented Apr 24, 2019

aschmahmann commented Apr 24, 2019

aschmahmann commented Apr 24, 2019 • edited Loading

aschmahmann Apr 24, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aschmahmann Apr 24, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aschmahmann Apr 24, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vyzo commented Mar 23, 2020

aschmahmann commented Apr 24, 2019 •

edited

Loading

aschmahmann Apr 24, 2019 •

edited

Loading

aschmahmann Apr 24, 2019 •

edited

Loading

aschmahmann Apr 24, 2019 •

edited

Loading