feat: add HTTP Gateway support for /ipns/ paths #2020

lidel · 2019-05-07T15:13:36Z

Part of an effort to run embedded js-ipfs in Brave 🦁 ipfs/ipfs-companion#716
Fixes #1918

This PR will add support for /ipns/ paths at HTTP Gateway.
Smoke test: /ipns/tr.wikipedia-on-ipfs.org (IPNS+DNSLink+HAMT-sharded website)

Do not merge it yet

This PR requires below PRs to be merged first:

Gateway Improvements from Gateway Improvements: Streaming, Conditional and Range Requests #1989
(after merging Gateway Improvements: Streaming, Conditional and Range Requests #1989 I will rebase this PR, which will remove first two commits)
PeerID
eg. /ipns/<PeerId-as-multihash-b58>
- requires feat: add support for ipns name resolve /ipns/<fqdn> #2002 to land first
/ipns/<libp2p-key-in-cidv1>
- requires feat: add libp2p-key and garlic32 to js-multicodec multiformats/js-multicodec#45
DNSLink
eg. /ipns/<fqdn>/path/file like /ipns/docs.ipfs.io/assets/logo.svg
- requires feat: add support for ipns name resolve /ipns/<fqdn> #2002 to land first
HAMT shard support
eg. /ipns/tr.wikipedia-on-ipfs.org/wiki/Anasayfa.html (wiki is a sharded directory)
- requires Sharding support in /ipns/<fqdn> paths js-ipfs-http-response#22 and feat: handle raw nodes in mfs ipfs-inactive/js-ipfs-mfs#48 to land first
Tests for /ipns/

tl;dr opening /ipns/tr.wikipedia-on-ipfs.org/wiki/Mars.html works Switched to js-ipfs with cherry-picked changes from: ipfs/js-ipfs#2020 ipfs/js-ipfs#1989 ipfs/js-ipfs#2002 ipfs/js-ipfs-http-response#22 ipfs-inactive/js-ipfs-mfs#48

It requires below to PRs to land first: #2002 ipfs/js-ipfs-http-response#19 ipfs-inactive/js-ipfs-mfs#48 License: MIT Signed-off-by: Marcin Rataj <lidel@lidel.org>

lidel · 2019-06-27T07:26:42Z

I rebased this PR and switched to released versions of ipfs-mfs and ipfs-http-response.
Cool test link for HTTP Gateway (IPNS+DNSLink+HAMT):

http://127.0.0.1:9090/ipns/tr.wikipedia-on-ipfs.org/wiki/Anasayfa.html

Note: This PR resolves /ipns/ to /ipfs/ before passing it to ipfs-http-response as a workaround until ipfs/js-ipfs-http-response#22 is solved in more generic way.

@alanshaw this is a fairly small change, do you want to merge this to compliment other IPNS-related backend PRs made by @hugomrdias?

(Travis fails due to recent changes in master, not related to this PR: https://travis-ci.com/ipfs/js-ipfs/jobs/211405524#L3104)

src/http/gateway/routes/index.js

src/http/gateway/resources/gateway.js

src/http/gateway/routes/gateway.js

hugomrdias · 2019-06-28T16:40:35Z

src/http/gateway/routes/gateway.js

+  },
+  {
+    method: '*',
+    path: '/ipns/{mutableId*}',


we should define what we should call this id, libp2p-key, ipns-key-id, even ipns-cid (content been the hash of the pubkey) and we should PR to the spec https://github.com/ipfs/specs/tree/master/naming this new name.

We have libp2p-key multicodec now for presenting IPNS names as base32 CIDv1, but it can also be FQDN with DNSLink, so I renamed it to libp2pKeyOrFqdn.

I did not touch mentioned spec as /ipns/base32(<HASH>) mentioned there has direct implication on how things are implemented: https://github.com/ipfs/js-ipns/blob/v0.5.2/src/index.js#L214 (IPNS does not use <libp2p-key-in-cidv1b32> but raw concatenation of /ipns/ and key buffer, and now they move to just buffer)

License: MIT Signed-off-by: Marcin Rataj <lidel@lidel.org>

- remove custom arg passing and replace it with native Joi validation - decode escaped unicode in paths before passing to ipfs resolver - add tests for /ipns/ path (file and directory) License: MIT Signed-off-by: Marcin Rataj <lidel@lidel.org>

License: MIT Signed-off-by: Marcin Rataj <lidel@lidel.org>

src/http/gateway/resources/gateway.js

License: MIT Signed-off-by: Marcin Rataj <lidel@lidel.org>

hugomrdias

LGTM

alanshaw

Code changes LGTM, just one question.

alanshaw · 2019-07-04T15:44:38Z

src/http/api/routes/webui.js

 const resources = require('../../gateway/resources')

 module.exports = [
  {
    method: '*',
-    path: '/ipfs/{cid*}',
+    path: '/ipfs/{path*}',


Does this need an /ipns handler also?

No, this is only for /webui and hardcoded /ipfs/{cid_of_webui} on API port.
(/ipns/ is only available on the Gateway port)

alanshaw · 2019-07-04T15:48:53Z

test/gateway/index.js

+    // QmaRdtkDark8TgXPdDczwBneadyF44JvFGbrKLTkmTUhHk
+    await http.api._ipfs.add([content('utf8/cat-with-óąśśł-and-أعظم._.jpg')])
+    // Publish QmW2WQi7j6c7UgJTarActp7tDNikE4B2qXtFCfLPdsgaTQ to IPNS using self key
+    await http.api._ipfs.name.publish('QmW2WQi7j6c7UgJTarActp7tDNikE4B2qXtFCfLPdsgaTQ', { resolve: false })


Note for the future: unless we're trying to validate particular content creates a particular CID we should add the content and store the returned CID for use in future tests. Hard coding CIDs like this is a maintenance nightmare when we want to change defaults.

I am aware, but did not want to rewrite existing tests, so as a compromise I reused existing CID of cat-folder/cat.jpg (was already present a few lines above)

ghost assigned lidel May 7, 2019

ghost added the status/in-progress In progress label May 7, 2019

lidel changed the title ~~feat: add HTTP Gateway support for /ipns/ paths~~ [WIP] feat: add HTTP Gateway support for /ipns/ paths May 7, 2019

hugomrdias mentioned this pull request May 7, 2019

IPNS over DNS #2000

Closed

15 tasks

lidel mentioned this pull request May 8, 2019

feat(brave): Embedded HTTP Gateway for /ipns/{fqdn} with sharding ipfs/ipfs-companion#719

Merged

lidel force-pushed the feat/ipns-on-gateway branch from 571eb4b to c9fe577 Compare May 13, 2019 11:59

feat: support /ipns/ at HTTP Gateway

97d03fc

It requires below to PRs to land first: #2002 ipfs/js-ipfs-http-response#19 ipfs-inactive/js-ipfs-mfs#48 License: MIT Signed-off-by: Marcin Rataj <lidel@lidel.org>

lidel force-pushed the feat/ipns-on-gateway branch from c9fe577 to 97d03fc Compare June 27, 2019 07:21

lidel changed the title ~~[WIP] feat: add HTTP Gateway support for /ipns/ paths~~ feat: add HTTP Gateway support for /ipns/ paths Jun 27, 2019

lidel requested review from alanshaw and hugomrdias June 27, 2019 07:30

hugomrdias suggested changes Jun 28, 2019

View reviewed changes

lidel added 4 commits July 2, 2019 21:12

Merge master into feat/ipns-on-gateway

31233e0

License: MIT Signed-off-by: Marcin Rataj <lidel@lidel.org>

style: apply name suggestions from review

99d92b7

License: MIT Signed-off-by: Marcin Rataj <lidel@lidel.org>

style: unify route param names

eebde0b

License: MIT Signed-off-by: Marcin Rataj <lidel@lidel.org>

hugomrdias suggested changes Jul 3, 2019

View reviewed changes

lidel added 3 commits July 3, 2019 17:26

test(gateway): load from URI-encoded path

c21709d

License: MIT Signed-off-by: Marcin Rataj <lidel@lidel.org>

style: unify names related to content path

7e3e7b2

License: MIT Signed-off-by: Marcin Rataj <lidel@lidel.org>

style: immutablePath → ipfsPath

f1da001

License: MIT Signed-off-by: Marcin Rataj <lidel@lidel.org>

hugomrdias approved these changes Jul 3, 2019

View reviewed changes

alanshaw approved these changes Jul 4, 2019

View reviewed changes

alanshaw mentioned this pull request Jul 4, 2019

⚡️ v0.37.0 RELEASE 🚀 #2192

Closed

54 tasks

alanshaw merged commit 43ac305 into master Jul 4, 2019

alanshaw deleted the feat/ipns-on-gateway branch July 4, 2019 16:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add HTTP Gateway support for /ipns/ paths #2020

feat: add HTTP Gateway support for /ipns/ paths #2020

lidel commented May 7, 2019 •

edited

Loading

lidel commented Jun 27, 2019 •

edited

Loading

hugomrdias Jun 28, 2019

lidel Jul 3, 2019 •

edited

Loading

hugomrdias left a comment

alanshaw left a comment

alanshaw Jul 4, 2019

lidel Jul 4, 2019 •

edited

Loading

alanshaw Jul 4, 2019

lidel Jul 4, 2019 •

edited

Loading

feat: add HTTP Gateway support for /ipns/ paths #2020

feat: add HTTP Gateway support for /ipns/ paths #2020

Conversation

lidel commented May 7, 2019 • edited Loading

Do not merge it yet

lidel commented Jun 27, 2019 • edited Loading

hugomrdias Jun 28, 2019

Choose a reason for hiding this comment

lidel Jul 3, 2019 • edited Loading

Choose a reason for hiding this comment

hugomrdias left a comment

Choose a reason for hiding this comment

alanshaw left a comment

Choose a reason for hiding this comment

alanshaw Jul 4, 2019

Choose a reason for hiding this comment

lidel Jul 4, 2019 • edited Loading

Choose a reason for hiding this comment

alanshaw Jul 4, 2019

Choose a reason for hiding this comment

lidel Jul 4, 2019 • edited Loading

Choose a reason for hiding this comment

lidel commented May 7, 2019 •

edited

Loading

lidel commented Jun 27, 2019 •

edited

Loading

lidel Jul 3, 2019 •

edited

Loading

lidel Jul 4, 2019 •

edited

Loading

lidel Jul 4, 2019 •

edited

Loading