test: add ipns pubsub tests #39

vasco-santos · 2018-10-27T23:01:10Z

PR containing a set of tests aimimg to test the IPNS over Pubsub interoperability between go-ipfs and js-ipfs.

It needs the following:

go-ipfs released with libp2p/go-libp2p-pubsub-router#17
js-ipfs released with ipfs/js-ipfs/pull/1559

vasco-santos · 2018-10-27T23:02:14Z

Currently, we are facing a problem regarding the interop of the pubsub topic.

go-ipfs sets the topic as /ipns/BINARY_PEER_ID. In Golang strings are a slice of bytes, and consequently, do not have a specific encoding. On the other side, strings in Javascript must have an encoding format associated. This way, when decoding /ipns/BINARY_PEER_ID to the topic string and sending the message to a go peer, it cannot parse the topic correctly.

Considering all this and that IPNS over Pubsub is still an experimental feature, I have been talking with @Stebalien about changing the topic format to a standard encoding, such as base64 for both implementations.

Stebalien · 2018-10-31T19:45:15Z

So, some history: we used to use /ipns/Qm... but I changed it to /ipns/BINARY_ID to match what we do in the DHT. This allows us to treat pubsub as just another (kind of funky) record store (see https://github.com/libp2p/go-libp2p-pubsub-router). This, in turn, means we don't need to duplicate all the IPNS logic for both the DHT and Pubsub (or any other "record stores" we make in the future.

However, given that record-store keys are binary (arguably, they should also just be strings...), we could use /rs/base32(key) (or base64url). In this case, that would be /record/$base64url(/ipns/BINARY_KEY).

However, I'd also like to consider just making pubsub topics arbitrary binary strings. We already do this for the DHT and forcing users to use UTF-8 for pubsub topics will likely cause issues like this in the future as well (some implementations will enforce it, others won't).

Stebalien · 2018-10-31T19:55:17Z

I've hashed this out with @vyzo on IRC. He agrees we should probably be nice and use UTF8.

So, the only remaining question is: How? Personally, I'd vote for either /record/base32(...) or /record/base64url(...). Thoughts?

vasco-santos · 2018-11-02T11:04:48Z

Thanks for your analysis @Stebalien .

I would go with /record/base64url(/ipns/BINARY_KEY) since it will need fewer bytes and the topic will go with every single message. In addition, with ipns.pubsub.subs we can get the decoded topic.

Instead of using bare keys (binary), use `/record/base64url(key)`. This: 1. Pubsub keys must be UTF-8. Record-store keys definitely are not. 2. The prefix may help with validators in the future (namespace *everything*). See: ipfs/interop#39

Stebalien · 2018-11-15T02:53:02Z

@vyzo suggested base32 to be consistent with everything else we do. However, I really don't mind either way. To get this ball rolling: libp2p/go-libp2p-pubsub-router#17

magik6k · 2018-11-15T04:34:13Z

I'd use base64 since this is internal, and we send these topics to all peers we are pubsubbing with, so b64 saves us a few bytes each time we connect to someone.

vasco-santos · 2018-11-22T15:50:19Z

Interop between go-ipfs and js-ipfs is green locally using:

JS: ipfs/js-ipfs#1559 with ipfs/js-datastore-pubsub/pull/9 for fixing the interop

Go: go-ipfs#master with libp2p/go-libp2p-pubsub-router#17 (binary go-ipfs can be obtained with: https://ipfs.io/ipfs/QmXQDCKdsEaeDyejWn2ziXfTSjBd5YEZ6zoHTdrpMJWyvf)

@Stebalien do you have any eta for the new go-ipfs release with this?

@alanshaw maybe we can add this to v0.34.0 RELEASE 🚀

test/utils/wait-for.js

alanshaw · 2019-01-02T10:07:23Z

test/ipns-pubsub.js

+  before(function (done) {
+    this.timeout(100 * 1000)
+
+    parallel([


Suggestion: async/each

I think it is better to use async/parallel here, since using async/each here would require an extra array to keep the data with all the ids, which I feel unnecessary just for 2 elements.

alanshaw · 2019-01-02T10:09:16Z

test/ipns-pubsub.js

+      nodeBId = ids[1]
+
+      nodes[0].api.swarm.connect(ids[1].addresses[0], () => {
+        setTimeout(done, 60000) // wait for republish as we can receive the republish message first


This is such a long wait, it would be great if this was somehow configurable...

@Stebalien is it possible in go-ipfs?

@vasco-santos perhaps add a console.log line here to inform the user this pause is happening...?

go-ipfs already accepts the republish configuration through the node config. For JS Land, it is one of the things that I want to integrate during this quarter.

Meanwhile, I can add a console.log here!

Oh wait, go-ipfs allows the RepublishPeriod and RecordLifetime, but not the time for the first InitialRebroadcastDelay, afaik!

test/ipns-pubsub.js

daviddias · 2019-02-13T09:54:57Z

@vasco-santos mind rebasing master onto this PR and test it with js-ipfs master?

alanshaw · 2019-04-15T12:22:21Z

@vasco-santos what's the status of this? Can we get it merged plz 🙏?

vasco-santos · 2019-04-17T11:34:21Z

Subscribe in js and Publishing in go is currently failing. It seems that js node is not receiving the pubsub message once go publishes.

I just tested this using the latest js-ipfs and go-ipfs CLIs and everything worked fine, so I will debug this once I have some time

vasco-santos · 2019-05-13T11:31:24Z

@alanshaw it is 💚 🎊

ghost assigned vasco-santos Oct 27, 2018

ghost added the status/in-progress In progress label Oct 27, 2018

vasco-santos added the status/blocked Unable to be worked further until needs are met label Nov 4, 2018

vasco-santos force-pushed the add-ipns-pubsub-tests branch from 5c46bc3 to 9b9de01 Compare November 5, 2018 09:56

This was referenced Nov 5, 2018

docs: ipns ipfs/specs#184

Merged

refactor: ipns routing logic moved to instantiation ipfs/js-ipfs#1701

Merged

vasco-santos force-pushed the add-ipns-pubsub-tests branch from 9b9de01 to 9e55042 Compare November 11, 2018 15:17

vasco-santos mentioned this pull request Nov 14, 2018

Implement IPNS ipfs/js-ipfs#209

Closed

Stebalien mentioned this pull request Nov 15, 2018

encode record-store keys in pubsub libp2p/go-libp2p-pubsub-router#17

Merged

vasco-santos mentioned this pull request Nov 20, 2018

feat: ipns over pubsub ipfs/js-ipfs#1559

Merged

7 tasks

vasco-santos force-pushed the add-ipns-pubsub-tests branch from 9e55042 to 68592aa Compare November 22, 2018 15:43

vasco-santos mentioned this pull request Nov 22, 2018

feat: encode record-store keys in pubsub ipfs/js-datastore-pubsub#9

Merged

vasco-santos changed the title ~~[WIP] test: add ipns pubsub tests~~ test: add ipns pubsub tests Nov 26, 2018

vasco-santos force-pushed the add-ipns-pubsub-tests branch from 68592aa to 3c0f3e1 Compare November 26, 2018 15:06

vasco-santos mentioned this pull request Dec 3, 2018

fix: tests using files #48

Closed

1 task

alanshaw suggested changes Jan 2, 2019

View reviewed changes

BazaarDog mentioned this pull request Jan 3, 2019

Awesome Endeavour: DHT Part II ipfs/js-ipfs#856

Merged

35 tasks

vasco-santos force-pushed the add-ipns-pubsub-tests branch from 48c5089 to abdb8b3 Compare January 10, 2019 10:43

daviddias changed the title ~~test: add ipns pubsub tests~~ [WIP] test: add ipns pubsub tests Feb 13, 2019

vasco-santos force-pushed the add-ipns-pubsub-tests branch from abdb8b3 to 7fef266 Compare April 16, 2019 17:01

vasco-santos force-pushed the add-ipns-pubsub-tests branch 3 times, most recently from 48f1bc6 to 2353dbc Compare April 17, 2019 10:21

vasco-santos added 2 commits May 13, 2019 11:32

test: add ipns pubsub tests

3f43521

fix: code review

d63ee61

vasco-santos force-pushed the add-ipns-pubsub-tests branch from 2353dbc to d63ee61 Compare May 13, 2019 10:44

vasco-santos changed the title ~~[WIP] test: add ipns pubsub tests~~ test: add ipns pubsub tests May 13, 2019

alanshaw approved these changes May 13, 2019

View reviewed changes

alanshaw merged commit 3e89ad4 into master May 13, 2019

alanshaw deleted the add-ipns-pubsub-tests branch May 13, 2019 11:35

whyrusleeping removed the status/in-progress In progress label May 13, 2019

aschmahmann mentioned this pull request Jun 24, 2020

ipns over pubsub: howto? aschmahmann/ipns-utils#1

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: add ipns pubsub tests #39

test: add ipns pubsub tests #39

vasco-santos commented Oct 27, 2018 •

edited

Loading

vasco-santos commented Oct 27, 2018

Stebalien commented Oct 31, 2018

Stebalien commented Oct 31, 2018

vasco-santos commented Nov 2, 2018

Stebalien commented Nov 15, 2018

magik6k commented Nov 15, 2018

vasco-santos commented Nov 22, 2018

alanshaw Jan 2, 2019

vasco-santos Jan 2, 2019

alanshaw Jan 2, 2019

vasco-santos Jan 2, 2019

vasco-santos Jan 2, 2019

daviddias commented Feb 13, 2019

alanshaw commented Apr 15, 2019 •

edited

Loading

vasco-santos commented Apr 17, 2019

vasco-santos commented May 13, 2019

test: add ipns pubsub tests #39

test: add ipns pubsub tests #39

Conversation

vasco-santos commented Oct 27, 2018 • edited Loading

vasco-santos commented Oct 27, 2018

Stebalien commented Oct 31, 2018

Stebalien commented Oct 31, 2018

vasco-santos commented Nov 2, 2018

Stebalien commented Nov 15, 2018

magik6k commented Nov 15, 2018

vasco-santos commented Nov 22, 2018

alanshaw Jan 2, 2019

Choose a reason for hiding this comment

vasco-santos Jan 2, 2019

Choose a reason for hiding this comment

alanshaw Jan 2, 2019

Choose a reason for hiding this comment

vasco-santos Jan 2, 2019

Choose a reason for hiding this comment

vasco-santos Jan 2, 2019

Choose a reason for hiding this comment

daviddias commented Feb 13, 2019

alanshaw commented Apr 15, 2019 • edited Loading

vasco-santos commented Apr 17, 2019

vasco-santos commented May 13, 2019

vasco-santos commented Oct 27, 2018 •

edited

Loading

alanshaw commented Apr 15, 2019 •

edited

Loading