IPLD requirements for js-ipfs

[mikeal]

The IPLD team is trying to get a handle on our roadmap and plan for the next few quarters. js-ipfs is the primary consumer of JS IPLD interfaces and we’d like to get a good list of things js-ipfs needs over the next year or so.

[daviddias]

IPLD packs a ton of promises for the future of building distributed computing. However, rather than giving a long list of all future hopes and dreams, I'll make one request that reflects one current use case of IPLD over js-ipfs other than IPFS using it for UnixFS.

Can js-ipld reproducibility and correctly resolve data in formats used by blockchains such as the Ethereum Blockchain? This was possible before and it created a ton of momentum and value, things to note are:

• We made an example with it https://github.com/ipfs/js-ipfs/tree/master/examples/explore-ethereum-blockchain
• We made a huge demo -- https://www.youtube.com/watch?v=HxueJbeMVG4 --. It enphasises libp2p but the reality is that it was only possible thanks to IPLD
• Took notes on next steps IPFS & libp2p <3 Ethereum - Roadmap of things to get everything working together nicely 👌  notes#173
• Ran a call for a while 🦑 IPLD deep dive calls ⚓️ team-mgmt#484
• Got a bridge from Ethereum to IPFS working using the Parity client -- 🌟 Parity to IPFS #763
• A new project, Mustekala, got funded and developed https://medium.com/metamask/metamask-labs-presents-mustekala-the-light-client-that-seeds-data-full-nodes-vs-light-clients-3bc785307ef5.

It would be really great to have this stable and something that we can point people at as a great example of interoperability

[daviddias]

Ah, another great example of interoperability between networks and formats is https://github.com/ipfs-shipyard/igis, right now it lacks the maintenance necessary.

[daviddias]

One more ask (😇) support for Torrents #779

[mikeal]

torrents

I actually thought that torrent codecs were already implemented because, at some point, you added repos for those implementations to https://github.com/ipld/ipld#implementations but apparently those are just empty repos 😱

Writing JS codecs for torrents would be pretty easy, I’ve spent time a fair amount of time in those torrent libraries already. I’m not going to consider this high priority (compared to unixfsv2) but this is something small and fun and I’ll probably get to it in Q4 cause I love you @daviddias 😘

ETH

AFAIK the codecs for ETH work and aren’t in need of any dramatic changes (@vmx already did a big refactor recently that puts them in alignment with our other async/await refactors).

Our team is pretty small and focused on improvements to the tools and primitives, this makes taking the “ETH on IPFS demo” and turning it into a longer term supported application quite difficult for our team to do directly. However, if there’s anything this demo needs from IPLD libraries we can prioritize those more highly, just let us know what it is.

ETH2 is another story and we’re following that development and even in the process of providing feedback to some of the ETH folks on one of their proposed new block formats.

In general, our focus this year has been on IPLD Schemas and IPLD “native” block formats and workflows. But it’s important that we don’t forget about the basics, like just being able to link between block formats, and when we do planning for 2020 we should consider taking on any new block formats and even making some usability improvements to existing formats like git (which works but at a very low level, we could do more here to make it easier to pull out actual file data).

[daviddias]

you added repos for those implementations to https://github.com/ipld/ipld#implementations but apparently those are just empty repos 😱

Ups, sorry! I did a lot of experimentation locally and found some obstacles along the way, then my attention was routed to other things (it was mid 2017 :)). I left a list of things to be done at #779

Writing JS codecs for torrents would be pretty easy, I’ve spent time a fair amount of time in those torrent libraries already.

True, but depending on how it is integrated, it gets tricky. A torrent file is referenced by the hash (called infohash in torrent world) of the file that the torrent is sharing, but not the torrent file itself. torrent files keep mutating as new trackers get added.

Adding support to resolve torrent file chunks to IPLD should be trivial, I agree, however, in the ideal world one should be able to do something like ipfs torrent cat /infohash and it would pipe out the contents of the torrent by connecting to the torrent network and piecing the file together with bits existing there and other bits already existing in the IPFS network, if available.

I’m not going to consider this high priority (compared to unixfsv2) but this is something small and fun and I’ll probably get to it in Q4 cause I love you @daviddias 😘

I didn't mean to derail the plan for Q4, but this does make me super happy, thaaank you ❤!

ETH

Understood. It would be good to:

• Ensure that the example is running perfectly
• Potentially re record a video demo that can be shared
• Reach out to @kumavis and @hermanjunge to learn what is the latest state of Mustekala. I recall that for the Blockchain usecase, Bitswap was not ideal and now with Graphsync, it can potentially make the use case much more powerful.

[mikeal]

True, but depending on how it is integrated, it gets tricky. A torrent file is referenced by the hash (called infohash in torrent world) of the file that the torrent is sharing, but not the torrent file itself. torrent files keep mutating as new trackers get added.

Ya, I had read the prior threads and I’m already taking for granted the torrent-file + torrent approach at the IPLD layer.

Adding support to resolve torrent file chunks to IPLD should be trivial, I agree, however, in the ideal world one should be able to do something like ipfs torrent cat /infohash and it would pipe out the contents of the torrent by connecting to the torrent network and piecing the file together with bits existing there and other bits already existing in the IPFS network, if available.

As a first step, we should just get ipfs cat CID-of-torrent-file working.

Rather than infohash, why don’t you just use a magnet URL? We could even add a parameter to the magnet URL for the CID of the torrent-file, then it would contain everything we need for getting the file out of both networks. Maybe this sounds like cheating, but from a user perspective it’s already common to prefer magnet urls over infohashes, and you’ve already stubbed out a separate command in your example for getting torrents so the parameter doesn’t have to conform to multiaddr.

The networking side of this is outside the scope of IPLD but I can try to advise where I can since I have some experience with these JS libraries and with bittorrent in general.

[daviddias]

Two other requirements:

• The need for a "patch" like API IPLD Resolver patch API ipld/js-ipld#66 (and API optimized for applications of IPLD that need to do vast amounts of operations and can't afford to create a dag node for everything)
• Land a spec for the DAG API that is inclusive of the problems created when doing things over HTTP https://github.com/ipfs/interface-js-ipfs-core/issues/81

[mikeal]

I’ve thought a lot about the “patch” API and I’m still not sure it’s possible to do something atomic and performant.

A patch API, or any mutation API, needs to be efficiently batched in order to be performant when manipulating the same parts of the DAG concurrently. This is actually something I plan on writing soon on top of unixfsv2 (for mfs), taking all concurrent mutation operations and batching them into one large graph mutation that doesn’t unnecessarily create/read any part of the tree more than once.

I can produce this as a standalone library for efficient graph manipulation, but the API will have to sit on top of a larger mutex used for batching concurrent operations and I’m not sure how well that would fit into js-ipfs’s current architecture. It would have an API along the lines of:

const m = mutex(rootNode, { /* storage functions */ getBlock, setBlocks })
await m.set(‘/path/to/node’, newBlock)

You could use this atomically by just creating a new mutex for every operation, but it won’t be nearly as performant as keeping a single instance around and hammering it with concurrent mutations like I’ll do with the mfs code.

In short, you’ll have something for this by the end of the quarter, but it may take larger changes in js-ipfs in order to make it actually perform well.

[SgtPooki]

js-ipfs is being deprecated in favor of Helia. You can learn more about this deprecation and read the migration guide.

Please feel to reopen with any comments by 2023-06-02. We will do a final pass on reopened issues afterwards (see #4336).