Implement the core federation protocol #2491

masih · 2024-01-23T18:05:21Z

First pass at implementing the IPNI federation protocol:

core data structures/functionality
periodic snapshot
basic vector clock reconciliation
HTTP serve mux for /ipni/v1/fed

codecov-commenter · 2024-01-23T18:10:00Z

Codecov Report

Attention: 281 lines in your changes are missing coverage. Please review.

Comparison is base (97dcd6b) 47.02% compared to head (15c071d) 46.88%.
Report is 4 commits behind head on main.

Files	Patch %	Lines
federation/federation.go	33.69%	171 Missing and 14 partials ⚠️
federation/handlers.go	27.77%	32 Missing and 7 partials ⚠️
federation/schema.go	61.25%	23 Missing and 8 partials ⚠️
federation/options.go	69.13%	21 Missing and 4 partials ⚠️
federation/vector_clock.go	93.33%	1 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2491      +/-   ##
==========================================
- Coverage   47.02%   46.88%   -0.14%     
==========================================
  Files          85       90       +5     
  Lines       10718    11242     +524     
==========================================
+ Hits         5040     5271     +231     
- Misses       5079     5337     +258     
- Partials      599      634      +35

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

willscott · 2024-01-23T18:09:37Z

federation/federation.go

+	}, nil
+}
+
+func (f *Federation) Start(_ context.Context) error {


Why take and not use cancel here?

Because it might be surprising to the user that start context is tied to the entire lifecycle of federation rather than just starting it.

federation/federation.go

willscott · 2024-01-23T18:20:23Z

federation/schema.go

+		link ipld.LinkPrototype
+	}
+
+	//go:embed schema.ipldsch


This is an interesting pattern - does it impact startup time to do this on startup rather than a compile time step?

I have not measured it. I also have not heard complains about its current usage elsewhere being slow.

Happy to measure it if you think it's a blocker.

federation/schema.ipldsch

gammazero

There are a few minor coding issues commented on, but the more important thing is to establish an understanding of how to do:

Discovery of new indexers within the federation.
Authentication to allow membership
Reconsiliation process.

As for discovery, an indexer could be given a list of other indexer peer IDs. Or, indexers could periodically announce their presence on a libp2p topic, where the topic is the name of the federation. Other indexers subscribed to that same topic/federation whould choose whether to accept another indexer as a federation peer.

If discovery is limited to a configured list, where is this list published? If using libp2p pubsub, is there some bootstrap node somewhere? Is there a registry of "trusted" indexers for a given federation?

federation/federation.go

federation/handlers.go

gammazero · 2024-01-24T01:58:59Z

federation/schema.go

+		Epoch       uint64
+		VectorClock uint64
+		Providers   ProvidersIngestStatusMap
+		Previous    ipld.Link


In what, if any, situations will a previous snapshot be asked for by another indexer? A previous snapshot does not seem necessary for reconciliation.

The only situation I can see needing a previous snapshot for, is when an ipni instance (ipni-A) sees that another instance (ipni-B) has a provider unknown to ipni-A. Then ipni-A might look at previous snapshots to see if it used to know about the provider which was later delisted, or whether the provider is new to ipni-A.

A previous snapshot does not seem necessary for reconciliation.

Right.

The rationale for having a chain of snapshot is to provide opportunity for reputation measurements in the future since IPNI will need some sort of reputation system atop a permissionless federation setup.

Side note: It might be worth bringing up this question on the spec PR as it relates more to protocol design itself than the implementation. I think it is worth writing a rationale for its existence in the spec at least, if we reach the consensus to keep it.

gammazero · 2024-01-24T02:40:35Z

federation/federation.go

+func (f *Federation) reconcile(ctx context.Context, t time.Time) error {
+	// TODO: For each f.peers, get head, get snapshot, f.vc.reconcile, aggregate providers.
+	return nil
+}


Given snapshots:

snap-ipni-A (head) -------------------- Provider-P1 lastAd=10 Provider-P2 lastAd=5 Prev ---+ | -------------------- Provider-P1 lastAd=3 Provider-P2 lastAd=4 snap-ipni-B (head) -------------------- Provider-P1 lastAd=6 Provider-P2 lastAd=11 Prev ---+ | -------------------- Provider-P1 lastAd=4 Provider-P2 lastAd=3

I think the algorithm works something like:
ipni-B reconciles P1:

Get head ad for P1 (15)

If no vector clock difference for P1

Traverse P1 ad chain until ipni-A-P1-lastAd(10) or ipni-B-P1-lastAd(6)

Reconcile vector clocks for P1 and compute difference

If vector clock difference for P1 known:

Determine if ipni-A is ahead in indexing of P2

See that ipni-A is indexed farther, so sync P1 using ipni-A up to ad 10

ipni-B reconciles P2:

Get head ad for P2 (12)

If no vector clock difference for P2

Traverse P2 ad chain until ipni-A-P2-lastAd(11) or ipni-B-P1-lastAd(5)

Reconcile vector clocks for P2 and compute difference

If vector clock difference for P2 known:

Determine if ipni-A is ahead in indexing of P2

See that ipni-B is indexed farther, nothing to do

NOTE:

Previous snapshots are not examined in these situations. Maybe only when a provider is known to one ipni and not the other. Is this right?

The vector clock is only an optimization, and can only be used once an initial difference is established. I assume the clocks always accumulate and do not reset in each snapshot. Correct?

Right.

Correct. Vector clocks are monotonically increasing.

masih · 2024-01-24T10:25:19Z

establish an understanding of how to do:

Discovery of new indexers within the federation.

Authentication to allow membership

Reconsiliation process.

As for discovery, an indexer could be given a list of other indexer peer IDs. Or, indexers could periodically announce their presence on a libp2p topic, where the topic is the name of the federation. Other indexers subscribed to that same topic/federation whould choose whether to accept another indexer as a federation peer.

If discovery is limited to a configured list, where is this list published? If using libp2p pubsub, is there some bootstrap node somewhere? Is there a registry of "trusted" indexers for a given federation?

Discovery of indexers is strictly considered out of scope for the fist pass to 1) reduce scope, and 2) be continued as part of the ambient discovery proposal in IPFS specs. The list is configured as a static config when indexers start up, much like the bootstrap peers.
This makes the federation permission-ful in the initial implementation.
What can I help clarify re reconciliation process that's not covered by the spec?

First pass at implementing the IPNI federation protocol: * core data structures/functionality * periodic snapshot * basic vector clock reconciliation * HTTP serve mux for `/ipni/v1/fed`

willscott · 2024-01-30T21:06:44Z

federation/federation.go

+	mux.HandleFunc("/ipni/v1/fed/head", f.handleV1FedHead)
+	mux.HandleFunc("/ipni/v1/fed/", f.handleV1FedSubtree)


do we want this to be just /fed and then mount it within the /ipni/v1 general mux instead of parallel to it?

I strongly advise to have a unified prefix (plus version) for IPNI URIs. I think it makes it clear what the grouping is as a coherent protocol for content discovery.

Re mounting, the ingress reverse proxy would do the rewrite and it really makes no difference what is here. Except for the fact that the current mux makes it possible to run a fully self contained federation server separate from an indexer itself.

federation/federation.go

federation/vector_clock.go

Remove untick from vector clock at the risk of having increments larger than one for the local vector clock in edge cases where persistence of snapshot may fail in the midst of a cycle. This is an acceptable risk considering the difficulty in implementing transactional IPLD node storage and the application of it in this particular case.

It is still useful to have a head with no link to learn for example a federation member's public key or topic.

willscott reviewed Jan 23, 2024

View reviewed changes

gammazero reviewed Jan 24, 2024

View reviewed changes

Implement the core federation protocol

b049a7c

First pass at implementing the IPNI federation protocol: * core data structures/functionality * periodic snapshot * basic vector clock reconciliation * HTTP serve mux for `/ipni/v1/fed`

masih force-pushed the masih/federation branch from 8ca9196 to b049a7c Compare January 30, 2024 17:29

masih marked this pull request as ready for review January 30, 2024 17:29

willscott reviewed Jan 31, 2024

View reviewed changes

masih added 3 commits January 31, 2024 09:30

Add godoc to vector clock untick

8abfbba

Expand comments on epoch time dilution

3972590

masih force-pushed the masih/federation branch from dcd2af8 to 1e711a3 Compare January 31, 2024 09:47

masih added 3 commits January 31, 2024 11:41

Return no-content when there is no federation head

f2bcc6a

Make head link optional to cover bootstrap edge case

532e4f6

It is still useful to have a head with no link to learn for example a federation member's public key or topic.

Bump test timeout to 20 seconds

15c071d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement the core federation protocol #2491

Implement the core federation protocol #2491

masih commented Jan 23, 2024 •

edited

Loading

codecov-commenter commented Jan 23, 2024 •

edited

Loading

willscott Jan 23, 2024

masih Jan 24, 2024

willscott Jan 23, 2024

masih Jan 24, 2024

gammazero left a comment

gammazero Jan 24, 2024

masih Jan 24, 2024

gammazero Jan 24, 2024

masih Jan 24, 2024

masih commented Jan 24, 2024

willscott Jan 30, 2024

masih Jan 31, 2024

masih Jan 31, 2024

		mux.HandleFunc("/ipni/v1/fed/head", f.handleV1FedHead)
		mux.HandleFunc("/ipni/v1/fed/", f.handleV1FedSubtree)

Implement the core federation protocol #2491

Are you sure you want to change the base?

Implement the core federation protocol #2491

Conversation

masih commented Jan 23, 2024 • edited Loading

codecov-commenter commented Jan 23, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gammazero left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

masih commented Jan 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

masih commented Jan 23, 2024 •

edited

Loading

codecov-commenter commented Jan 23, 2024 •

edited

Loading