Story: Register a filesystem Directory in IPFS without Duplicating Content

[flyingzumwalt]

depends on #98: Use Manifest Files to Track Directory Structure & Contents

Story

Given:

I have a directory on my filesystem whose contents I want to serve through IPFS

Then:

I register the directory with IPFS, which turns it into an ipfs-pack. I will then be able to serve that content through IPFS by either running the ipfs-pack as a node or by relying on a local (intermediary) IPFS node.

Acceptance Scenario:

When a user tells ipfs to track a directory using ipfs add --nocopy

• IPFS builds an index of the content in that directory
• IPFS publishes the hashes for that content
• IPFS serves the blocks directly from the "tracked" files When remote nodes request content corresponding to those hashes

See notes from reviewing filestore

[kevina]

Please note that I do get notifications to every IPFS related issue. This directly involves me, yet I only discovered it by chance. Is there I reason I was not notified or mentioned?

[flyingzumwalt]

@kevina this sprint arose very quickly based on sudden interest outside our org. I'm setting up the issues & stories so we can discuss them in the Sprint Planning call. That planning call was scheduled for today but got moved to Tuesday because @whyrusleeping is traveling.

[kevina]

To add a directory with the what is in ipfs/kubo#3368 simply add a directory with the '--no-copy' option. This will pin the directory recursively like a normal add will. If the contents of the directory ever change than the entire directory can be readded. If the directory changes frequently than this will not work well.

The filestore was never designed to "track" a directory. I anticipated the situation where a directory changes frequently and wrote a script to add a directory in a way only changed files will need to be readded. The idea was that this script can be run at regular intervals in order to track changes. Unfortunately, doing it this way means that every file will need to be pinned individually, that can be slow and lead to a very large pinset. For this reason (and others) in the full filestore implementations ipfs/kubo#2634 all filestore objects are considered implicitly pinned. That means that the garbage collector will not remove anything in the filestore. A separate set of maintenance commands is provided to maintain the filestore.

[kevina]

Script link: https://github.com/ipfs-filestore/go-ipfs/tree/kevina/filestore/filestore/examples

Related issues: #93

[jbenet]

@kevina context: ipfs/team-mgmt#309 (comment)

[whyrusleeping]

Note that the filestore implementation satisfies the user story, but does not use the described internals. We won't be generating an ipfs-pack for each filestore added directory.

[flyingzumwalt]

ipfs add --nocopy -- docs in the new helptext feat/filestore0 PR ipfs/kubo#3629

[flyingzumwalt]

Removed this part of the Acceptance Scenario. This implementation doesn't use ipfs-pack. The ipfs-pack should be covered in the issues around that functionality.

How the Internals Should Work

When I tell IPFS to register a directory, it first turns the directory into an IPFS Pack. To do this, it

• builds an ipfs-pack manifest for the directory
• writes the ipfs-pack manifest in the root of the directory

It then registers the pack with your local ipfs node(s). To do this it

• generates an .ipfs repo in the root of the pack directory
• configures the repo to use filestore mode. meaning that it uses "relative paths" to resolve the content of its blocks.
• "adds" the ipfs-pack manifest to the repository's filestore

The structure of a pack is similar to git repositories -- the .ipfs directory sits in the root of the pack, alongside the other files/directories at the root of the pack. By default the relative path can only reference content inside of the pack -- just as git doesn't let you add content outside the repo.