initial implementation of save

[whyrusleeping]

I'm not sold on the name, or placement of this command, or even that this needs to be a command, or even that this is the right way to accomplish this (all of the above are up for discussion). But what I am set on is the fact that I need to be able to do what this command does.

[whyrusleeping]

Note: this does work with a live mounted fuse fs 😄

[wking]

So “Insert <QmSomeObject> as link <name> of <QmParentObject> and bubble changes up to <QmAncestorObject> …” seems like the general problem. I'd make re-publishing to IPNS a separate step. Something like: $ ipfs link set [--ancestor <ipfs-path>] <parent-ipfs-path> <name> <linkes-ipfs-path> spitting out the adjested parent ID if --ancestor wasn't set, or the adjusted ancestor ID if it was.

[wking]

More thoughts on this today on IRC from 1 through at least 2.
@tv42 pointed out that cases involving many edits should probably
queue writes to avoid writing nodes twice. For example, with a DAG
like:

a
|
b c

You might edit both (b) and (c) and just want to update (a) after both
of those changes.

Also, finding the path from a parent to a deeply-nested route is going
to be expensive if you're not tracking the link-path through the tree
while you traverse it. I don't think there's a good way around that
for a command-line UI, but supporting both:

$ ipfs link set [--ancestor ] <link's-ipfs-path>
$ ipfs link set [--ancestor ] /…// <link's-ipfs-path>

would allow the user to supply the path when they new it.

[whyrusleeping]

alright, i want to get back to working on this command.

I beleive the need for a staging area is apparent, so maybe we could do something git-like for this.

# start an edit under <root hash>
ipfs $CMD edit <root hash>

# add <hash> at <path> under the <root hash> for this edit session
ipfs $CMD add <path> <hash>

# delete <path> under <root hash>
ipfs $CMD rm <path>

# finalize this edit session and print out the modified root hash
ipfs $CMD commit

[wking]

On Sat, May 23, 2015 at 10:54:17AM -0700, Jeromy Johnson wrote:

start an edit under

ipfs $CMD edit

add at under the for this edit session

ipfs $CMD add

Hmm, this is going to be racy if you have multiple edits running at
once (e.g. you're developing two IPFS-versioned projects in parallel).
What are you getting by specifying the edit hash ahead of time? Maybe
we can find a non-racy way to achieve that. Alternatively, I guess we
could have optional edit-session IDs:

$ SESSION=$(ipfs $CMD edit $ROOT_HASH)
$ ipfs $CMD --session "${SESSION}" add $PATH $HASH
…

to override a default like “the most-recently opened session”.

[whyrusleeping]

@wking passing a root hash is used to specify your 'working dir' essentially, this command doesnt even make sense without it.

in regards to this being racy, using ids might be what we want to do. Doing things git style on the fielsystem doesnt make sense because clients will need to interface with this over the http API.

[jbenet]

Notes:

• I think the "edit session" idea is going to be very tricky to use and script with. the git staging area is different, because it's one per repo usually. We can do things like that, but i think editing an object is so common and simple that we may not need to do that.
• the word commit should be reserved for actual version control objects.
• the use of an env var to store the current edit is nice, but let's not keep "session state" in a node per se

What we need here is a plumbing command.

# return the hash of a <blank-root> object (0 links, 0 data)
> ipfs object blank  # or 'ipfs object new'
<blank-root>

# adds a link `name -> hash`
> ipfs object patch <root-hash> add-link <name> <hash>
<new-hash>

# removes all links with `name` or `hash` or `<int-offset>`
> ipfs object patch <root-hash> rm-link [ <name> | <hash> | <int-offset> ]
<new-hash>
# where <int-offset> is the number in the link table. 
# (<int-offset> is only way to address uniquely. names and hashes may repeat)

# set the data of the object. reads from stdin
> echo hello | ipfs object patch <root-hash> data
<new-hash>

These commands take the object, modify it, write it, and output the hash of the new object. writing an object is very cheap and will be gc-ed out (i.e. do not pin it automatically). pin can be a separate step.

a "session" turns into:

hello=$(echo "hello" | ipfs add -q)
world=$(echo "world" | ipfs add -q)

root=$(ipfs object blank)
root=$(ipfs object patch "$root" add-link world $world)
root=$(ipfs object patch "$root" add-link hello $hello)
root=$(ipfs object patch "$root" add-link world $world)
root=$(ipfs object patch "$root" rm-link 0)
echo $root

it's a bit annoying having to put the output of the previous command in the right place for the next, but we could:

• take <root> from stdin:

  ipfs object patch $root add-link hello $hello | ipfs object patch add-link world $world
  

• make a wrapping, porcelain command that uses the env-var:

# empty IPFS_PATCH_ROOT. is equivalent to <blank-root> (0 links, 0 data)

> ipfs patch $root add-link hello $hello   # exports IPFS_PATCH_ROOT=<new-root-1>
<new-root-1>

> ipfs patch $root add-link world $world   # exports IPFS_PATCH_ROOT=<new-root-2>
<new-root-2>

> ipfs patch clear # exports IPFS_PATCH_ROOT=<blank-root>

[jbenet]

(i'm hesitant to burn the top-level porcelain name ipfs patch on this, but pay be common enough to warrant it. or we could put the env-var functionality in the plumbing cmd too)

[whyrusleeping]

make a wrapping, porcelain command that uses the env-var:

sorry, thats not possible :/ best we can do is print out the new root

[jbenet]

sorry, thats not possible :/ best we can do is print out the new root

ah right :)

[wking]

On Thu, May 28, 2015 at 08:00:48AM -0700, Juan Batiz-Benet wrote:

What we need here is a plumbing command.

This non-bubbly form sounds reasonable to me, but I'd like an HTTP
endpoint and command-line command that wraps it to “add/replace/remove
from a/b/c and bubble the changes up to a”, since that seems like a
pretty common workflow.

return the hash of a object (0 links, 0 data)

ipfs object blank # or 'ipfs object new'

Maybe ‘empty’ instead of ‘blank’ or ‘new’?

adds a link name -> hash

ipfs object patch add-link

How is this going to extend to extensible links? See ipfs/ipfs#36 and
#1172. How do we manipulate the data field? Also, drilling down into
a multi-node layout (e.g. chunked file or fanned-out directory) is
going to be tedious (but I'm fine with that for plumbing).

removes all links with name or hash or <int-offset>

ipfs object patch rm-link [ | | ]

where is the number in the link table.

( is only way to address uniquely. names and hashes may repeat)

How do you distinguish the int-offset ‘0’ from the name ‘0’?

[whyrusleeping]

alright, so after implementing patch, doing what my save command here does is still very convoluted and difficult. Thoughts on actually merging this in one form or another?

[wking]

On Thu, May 28, 2015 at 07:52:22PM -0700, Jeromy Johnson wrote:

alright, so after implementing patch, doing what my save command
here does is still very convoluted and difficult. Thoughts on
actually merging this in one form or another?

First round of porcelain: immediate bubbling. Pardon the shell, and
this isn't as robust as a Go implementation could be (e.g. it doesn't
handle /ipfs/Qm…/… paths):

#!/bin/sh

usage: replace.sh IPFS_PATH ACTION REPLACMENT

For example:

replace.sh QmZKetgwm4o3LhNaoLSHv32wBhTwj9FBwAdSchDMKyFQEx/foo/bar/baz add-link QmTz3oc4gdpRMKP2sdGUPZTAGRngqjsi99BPoztyP53JMM

will add a 'baz' entry to

QmZKetgwm4o3LhNaoLSHv32wBhTwj9FBwAdSchDMKyFQEx/foo/bar pointing at

QmTz3oc4gdpRMKP2sdGUPZTAGRngqjsi99BPoztyP53JMM, and then update

QmZKetgwm4o3LhNaoLSHv32wBhTwj9FBwAdSchDMKyFQEx/foo to point to that,

and then print out the new root hash.

set -e

dereference an IPFS path

dereference() {
local IPFS_PATH="$1"
local CURRENT=$(echo "${IPFS_PATH}" | cut -d / -f 1)
local NEXT=$(echo "${IPFS_PATH}" | cut -d / -f 2)
local TAIL=$(echo "${IPFS_PATH}" | cut -d / -f 3-)
while test -n "${NEXT}"
do
CURRENT=$(ipfs object get "${CURRENT}" |
jq --raw-output ".Links[] | select(.Name == "${NEXT}").Hash")
NEXT=$(echo "${TAIL}" | cut -d / -f 1)
TAIL=$(echo "${TAIL}" | cut -s -d / -f 2-)
done
echo "${CURRENT}"
}

replace_link() {
local ROOT="$1"
local NAME="$2"
local VALUE="$3"
local REMOVED=$(ipfs patch "${ROOT}" rm-link "${NAME}")
ipfs patch "${REMOVED}" add-link "${NAME}" "${VALUE}"
}

replace() {
local IPFS_PATH="$1" # QmZKetgwm4o3LhNaoLSHv32wBhTwj9FBwAdSchDMKyFQEx/foo/bar/baz
local ACTION="$2" # add-link
local REPLACEMENT="$3" # QmTz3oc4gdpRMKP2sdGUPZTAGRngqjsi99BPoztyP53JMM
local CURRENT=$(echo "${IPFS_PATH}" | cut -d / -f 1) # QmZ...Ex
local NEXT=$(echo "${IPFS_PATH}" | cut -d / -f 2) # foo
local TAIL=$(echo "${IPFS_PATH}" | cut -d / -f 3-) # bar/baz
if test -n "${TAIL}"
then
local CHILD=$(dereference "${CURRENT}/${NEXT}")"/${TAIL}" # {hash-for-foo}/bar/baz
REPLACEMENT=$(replace "${CHILD}" "${ACTION}" "${REPLACEMENT}")
replace_link "${CURRENT}" "${NEXT}" "${REPLACEMENT}"
else
case "${ACTION}" in
add-link)
replace_link "${CURRENT}" "${NEXT}" "${REPLACEMENT}"
;;
rm-link)
ipfs patch "${CURRENT}" "${ACTION}" "${NEXT}"
;;
*)
echo "unrecognized action: '${ACTION}'" >&2
exit 1
esac
fi
}

replace "$@"

Making ‘dereference’ a builtin command (‘ipfs object stat’) is #1082.
I'd be open to a add-or-replace-link action that matches my
‘replace_link’ function. I'd like to move the new command in #1299 to
‘ipfs object patch’ and expose a Go version of ‘replace’ as ‘ipfs
patch’. Once we have all of that, we can get back to talking about
saved sessions with commits ;).

[jbenet]

alright, so after implementing patch, doing what my save command here does is still very convoluted and difficult. Thoughts on actually merging this in one form or another?

the patch porcelain on top of object patch?

@wking that's really good. 👍 on shell. we're on the git road ;) @chriscool would be proud. (i hope to be able to make prototyping command implementations in shell or other langs nicer before adding to api / core, etc).

@whyrusleeping said

Alternatively (or additionally) we could have an ipfs resolve command that outputs the hash that a path resolves to.

i like the

> ipfs resolve <path>
<hash>

idea (instead of dereference)

[wking]

On Mon, Jun 01, 2015 at 08:10:49PM -0700, Juan Batiz-Benet wrote:

Alternatively (or additionally) we could have an ipfs resolve
command that outputs the hash that a path resolves to.

i like the ipfs resolve <path> -> <hash> idea.

This is #1320 (and maybe also #1082), but I think we should think
through the intended command-line UI for resolution first (and that's
#1307). For example, ‘ipfs resolve …’ is a name that @jbenet doesn't
think should exist at all 1. But I'm not sure how that previous
comment squares with this new one ;).

[whyrusleeping]

maybe we could rename this to something like ipfs unixfs insert or something. But we cannot acheive the same functionality through the patch command or similar. Imagine the fuse mountpoint is mounted, and two different programs are editing different files at different places in the tree. using the patch command, if one of those writes to the dag, it breaks the entire tree that the other one is on. This command goes through the ipnsfs interface and is threadsafe, and wont break the fuse interface.

[wking]

On Thu, Jun 04, 2015 at 09:42:22AM -0700, Jeromy Johnson wrote:

Imagine the fuse mountpoint is mounted, and two different programs
are editing different files at different places in the tree. using
the patch command, if one of those writes to the dag, it breaks the
entire tree that the other one is on. This command goes through the
ipnsfs interface and is threadsafe, and wont break the fuse
interface.

Using the patch command shouldn't have any effect on the IPNS-linked
filesystem, since it's just creating new Merkle objects. The problem
seems to be the bug in publishing the new Merkle objects with ‘ipfs
name publish …’, and that's #1186.

[whyrusleeping]

closing for now, will reopen later when i get to it