-
Notifications
You must be signed in to change notification settings - Fork 1.2k
[WIP] feat: load IPLD formats lazily from IPFS #1830
Conversation
License: MIT Signed-off-by: Alan Shaw <alan.shaw@protocol.ai>
License: MIT Signed-off-by: Alan Shaw <alan.shaw@protocol.ai>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is really fantastic! This idea has been on the table for a while and it is great to see it implemented and working.
That said, I believe we need some sort of strategy to accommodate users that will be in a disjoint network (either not connected to main IPFS or not connected to main Internet at all). How will they be able to add these formats? Is there a config option to ensure that they get packaged up?
The IPLD Tutorial can/should be augmented to cover this.
What are the implications of running the test suite in the Browser? Would I need some additional step if I make a change to and IPLD Format? |
i would love to support this, load all things everywhere from ipfs thats the dream right ? but i can't right now ! there's still a long road ahead. here we are entering the dapps dev environment and devs, browsers, tooling still are based on http and others old and ugly tech :) imo doing this would bring rivers of issues and problems. my idea on the bundle-size PR is to give the dapp dev the same level of configuration for IPLD as they already have for libp2p, i.e. "i need zcash? yes! ok simple i require it and pass it in the ipfs config ipld prop. done!" small surface, easy to understand and explain, full flexibility. this proposal brings magic i.e. it just works but at what cost ?
we could try this behing an experimental flag to explore the idea and make sure we have some nice visible warnings to the dapp dev explaining what's happening (only in DEV environment process.env.NODE_ENV !== production) also we can explore this using plain http instead of ipfs var script = document.createElement('script');
script.onload = function () {
//do stuff with the script
};
script.src = 'url to ipls-git';
document.head.appendChild(script); still what would be a reliable url ? unpkg.com, ipfs gateway ? can we guarantee delivery ? |
I feel this is the future 🛰️, but like others mentioned, really rough around the edges today. Some thoughts about raised issues:
|
The consensus is that this will make the runtime non-deterministic so may result in unpredictable behaviour. However if this sort of thing is desired it can be done without code changes to core, you just need to know which CID a given codec name/code should map to: const { create } = require('ipfs-core')
const toBuffer = require('it-to-buffer')
const { toString } = require('uint8arrays/to-string')
const formatLookup = {
// resolves to a js file that implements a block codec for 'my-custom-format'
1337: CID.parse('bafyfoo')
}
const ipfs = await create({
ipld: {
async loadCodec (code) {
if (!formatLookup[code]) {
throw new Error('Unknown format ' + code)
}
const buf = await toBuffer(ipfs.cat(formatLookup[code]))
return eval(toString(buf))
}
}
})
// cause our custom format to be pulled from ipfs
const node = await ipfs.dag.get(CID.createV1(1337, digest)) |
I saw that @hugomrdias's PR #1795 will actually remove all the exotic IPLD formats from the browser build and at first I thought that was a good idea, but then I thought maybe not, and then I thought well maybe we can just do what @daviddias suggested in the first place and load the IPLD formats from IPFS.
This PR does that.
I created https://github.com/alanshaw/ipld-formats as a tool to get the browser builds of IPLD formats onto IPFS...and this PR uses the
loadFormat
option of IPLD to lazily load the module from IPFS. It simplyipfs.cat
s the module from a CID, runs the code and returns the module to IPLD.View this video for more info: https://youtu.be/ADA_HT2whok
In my local build I found the following bundle size improvement:
Before: 2,565,129 bytes (minified)
After: 1,864,080 bytes (minified)
~27% smaller
If y'all like the sound of this I'll pull the
ipfsJsLoader
into an external module and clean this up some more.I like that the bundle size is smaller but I don't like that the versions for these modules are now not in
package.json
and instead are in the code...I'm not sure what to do about that. Suggestions?cc @vmx @mikeal @ipfs/wg-gui-team