notes from a discussion between @whyrusleeping and @jbenet

[jbenet]

2016-01-16 discussion

Attendees:

• @whyrusleeping
• @jbenet

Location: Greenland, Canada
Altitude: 5km

go-ipfs notes

• fix unixfs importing permissions (metadata)
• ipfs get speed and ipfs stats bw --poll "Rate Down" discrepancy (~1 MB/s vs ~170 kB/s)
• should respond to (or periodically check for) network interface changes (+/- networks, etc).

Orbit notes

• UX issue: you can't get out of replying to a message once you click one.
• clocks: use the hash chain to figure out display order (vector clocks style), not the clock...
  we had clock drift, so one user's messages were coming in a few min behind. super confusing.
• Emojis, fonts, etc ALL should be distributed with Orbit. NEVER load from the internet.
• Chat window doesnt scroll down super well
• maybe have more colors for usernames. we have lots of collisions.
• #local channel: only your immediate, local network.
• presence: show list of users currently in the chat channel. gray dot = not connected, green dot = connected.
• add a light theme
• image extensions not tracked correctly for opening images in the chat
  • try "x.jpg" vs "x.JPG"
  • ".jpeg" does not work.
  • use a known-to-be-solid library for this.

data.gov

• @whyrusleeping has a diagram (post it here maybe?)
• improving add perf
• improving UX
• went over possible on-disk datastore changes
  • single mmapped file, btree index of offsets, unodered blocks after
• went over ipfs-pack, manifest, verify
  • how it combines very well with filestore
  • importer string
• other-repo-datastore
• accumulators wish list
• to discuss still:
  • s3-datastore
  • filestore implementation details

importers notes

• keep a table of importers (add to multicodec packed table), that uniquely define the function forever.
  • any change to the importer, including changing defaults, requires a new code (new version, effectively)
  • consider whether to use one varint for all versions, or a tuple <mcp-code-for-importer><importer-version>
• importer fmt string like <mbp><fmt-string-version-varint><mcp-code-for-importer><importer-args>
  • where <importer-args> can be like <hash-fn-multihash-code><dag-layout><chunker><chunk-size>
  • (i.e. a param can choose the chunker, we dont have to have different importers per chunker (otherwise combinations would explode)
• ipfs add -r --importer <importer-string> <dir> -- import based on a string, must determine whole graph.
• ipfs add -r --importer PackManifest <dir> -- import to match the given ipfs-pack manifest (use fmt strings there)
• default ipfs add importer
  • will change over time as sub-importers (like tar, mp4, zip, etc) are added
  • will change as default params may change (hash fn sha2 -> blake2b)
  • version the default importer, so that people can use old versions of the importer to reproduce old hashes
  • show version history in help text or elsewhere (importers.md?)
  • this, coupled with

mfs/reflog notes

• Consider adding all ipfs add added files to mfs somewhere, to help people track their files
• Want something like ~/Downloads folder
• Want something like a git reflog too. (may be different thing)
• Option A: add a /mfs/Added/ dir, and put all things there
  • pin /mfs/Added recursively, and switch default ipfs add --pin false
  • A.1 either all in that one folder.
    • oooof, messy. slow.
    • ok because HAMT? other things will choke though...
    • who will clean it up and delete refs? few people probably.
  • A.2 sharded by timestamp. /mfs/Added/YYYY/YYYY-MM-DD/<original-filename>
    • can easily find things based on time.
    • would help the importer formats stuff, because it would help people figure out what format string they must've added with.
• Option B: add an ipfs history that tracks all calls to ipfs
  • like history, but with
  • maybe with refs, like git reflog
  • can filter out to only get a set of options
  • coalesce last calls based on idempotency
  • can use filters to find specific actions (like, modifying state, or network related, under a subcommand, or containing some text)
    • or just use grep on top ;)
  • can probably set filters in the config on what to track/keep
  • need a good single-line format for how this would be tracked.
• Option C: keep a proper ipfs reflog of the HEAD of the ipfs repo
  • possible once ALL persistent ipfs repo state is captured in an ipfs dag
  • can walk back ref history to undo operations (like removing or adding pins)
  • make it easy to understand what changed? (ipfs diff?)
  • make it easy to explore the dag somehow? (like git log --graph --decorate ?)
• All these options ARE NOT mutually exclusive. they're all useful, and can coexist.

[jbenet]

cc @flyingzumwalt @whyrusleeping

[jbenet]

cc @haadcode

[rht]

fix unixfs importing permissions (metadata)

This is the bottleneck to having an archive file that can be used for the purpose of interop with NixOS's NAR file.

went over ipfs-pack, manifest, verify

• how it combines very well with filestore
• importer string

This task can be decoupled from the rest of scaling, UX tweaks, importer, mfs. If this is nailed first, small to mid datasets can be added.

[daviddias]

//cc @pgte - Importer / MFS notes