Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

initial implementation of save #1187

Closed
wants to merge 1 commit into from
Closed

initial implementation of save #1187

wants to merge 1 commit into from

Conversation

whyrusleeping
Copy link
Member

I'm not sold on the name, or placement of this command, or even that this needs to be a command, or even that this is the right way to accomplish this (all of the above are up for discussion). But what I am set on is the fact that I need to be able to do what this command does.

@whyrusleeping whyrusleeping added the status/in-progress In progress label May 2, 2015
@whyrusleeping
Copy link
Member Author

Note: this does work with a live mounted fuse fs 😄

@wking
Copy link
Contributor

wking commented May 2, 2015 via email

@wking
Copy link
Contributor

wking commented May 9, 2015

More thoughts on this today on IRC from 1 through at least 2.
@tv42 pointed out that cases involving many edits should probably
queue writes to avoid writing nodes twice. For example, with a DAG
like:

a
|
b c

You might edit both (b) and (c) and just want to update (a) after both
of those changes.

Also, finding the path from a parent to a deeply-nested route is going
to be expensive if you're not tracking the link-path through the tree
while you traverse it. I don't think there's a good way around that
for a command-line UI, but supporting both:

$ ipfs link set [--ancestor ] <link's-ipfs-path>
$ ipfs link set [--ancestor ] /…// <link's-ipfs-path>

would allow the user to supply the path when they new it.

@whyrusleeping
Copy link
Member Author

alright, i want to get back to working on this command.

I beleive the need for a staging area is apparent, so maybe we could do something git-like for this.

# start an edit under <root hash>
ipfs $CMD edit <root hash>

# add <hash> at <path> under the <root hash> for this edit session
ipfs $CMD add <path> <hash>

# delete <path> under <root hash>
ipfs $CMD rm <path>

# finalize this edit session and print out the modified root hash
ipfs $CMD commit

@wking
Copy link
Contributor

wking commented May 24, 2015

On Sat, May 23, 2015 at 10:54:17AM -0700, Jeromy Johnson wrote:

start an edit under

ipfs $CMD edit

add at under the for this edit session

ipfs $CMD add

Hmm, this is going to be racy if you have multiple edits running at
once (e.g. you're developing two IPFS-versioned projects in parallel).
What are you getting by specifying the edit hash ahead of time? Maybe
we can find a non-racy way to achieve that. Alternatively, I guess we
could have optional edit-session IDs:

$ SESSION=$(ipfs $CMD edit $ROOT_HASH)
$ ipfs $CMD --session "${SESSION}" add $PATH $HASH

to override a default like “the most-recently opened session”.

@whyrusleeping whyrusleeping mentioned this pull request May 26, 2015
49 tasks
@whyrusleeping
Copy link
Member Author

@wking passing a root hash is used to specify your 'working dir' essentially, this command doesnt even make sense without it.

in regards to this being racy, using ids might be what we want to do. Doing things git style on the fielsystem doesnt make sense because clients will need to interface with this over the http API.

@jbenet
Copy link
Member

jbenet commented May 28, 2015

Notes:

  • I think the "edit session" idea is going to be very tricky to use and script with. the git staging area is different, because it's one per repo usually. We can do things like that, but i think editing an object is so common and simple that we may not need to do that.
  • the word commit should be reserved for actual version control objects.
  • the use of an env var to store the current edit is nice, but let's not keep "session state" in a node per se

What we need here is a plumbing command.

# return the hash of a <blank-root> object (0 links, 0 data)
> ipfs object blank  # or 'ipfs object new'
<blank-root>

# adds a link `name -> hash`
> ipfs object patch <root-hash> add-link <name> <hash>
<new-hash>

# removes all links with `name` or `hash` or `<int-offset>`
> ipfs object patch <root-hash> rm-link [ <name> | <hash> | <int-offset> ]
<new-hash>
# where <int-offset> is the number in the link table. 
# (<int-offset> is only way to address uniquely. names and hashes may repeat)

# set the data of the object. reads from stdin
> echo hello | ipfs object patch <root-hash> data
<new-hash>

These commands take the object, modify it, write it, and output the hash of the new object. writing an object is very cheap and will be gc-ed out (i.e. do not pin it automatically). pin can be a separate step.

a "session" turns into:

hello=$(echo "hello" | ipfs add -q)
world=$(echo "world" | ipfs add -q)

root=$(ipfs object blank)
root=$(ipfs object patch "$root" add-link world $world)
root=$(ipfs object patch "$root" add-link hello $hello)
root=$(ipfs object patch "$root" add-link world $world)
root=$(ipfs object patch "$root" rm-link 0)
echo $root

it's a bit annoying having to put the output of the previous command in the right place for the next, but we could:

  • take <root> from stdin:

    ipfs object patch $root add-link hello $hello | ipfs object patch add-link world $world
    
  • make a wrapping, porcelain command that uses the env-var:

# empty IPFS_PATCH_ROOT. is equivalent to <blank-root> (0 links, 0 data)

> ipfs patch $root add-link hello $hello   # exports IPFS_PATCH_ROOT=<new-root-1>
<new-root-1>

> ipfs patch $root add-link world $world   # exports IPFS_PATCH_ROOT=<new-root-2>
<new-root-2>

> ipfs patch clear # exports IPFS_PATCH_ROOT=<blank-root>

@jbenet
Copy link
Member

jbenet commented May 28, 2015

(i'm hesitant to burn the top-level porcelain name ipfs patch on this, but pay be common enough to warrant it. or we could put the env-var functionality in the plumbing cmd too)

@whyrusleeping
Copy link
Member Author

make a wrapping, porcelain command that uses the env-var:

sorry, thats not possible :/ best we can do is print out the new root

@jbenet
Copy link
Member

jbenet commented May 28, 2015

sorry, thats not possible :/ best we can do is print out the new root

ah right :)

@wking
Copy link
Contributor

wking commented May 28, 2015

On Thu, May 28, 2015 at 08:00:48AM -0700, Juan Batiz-Benet wrote:

What we need here is a plumbing command.

This non-bubbly form sounds reasonable to me, but I'd like an HTTP
endpoint and command-line command that wraps it to “add/replace/remove
from a/b/c and bubble the changes up to a”, since that seems like a
pretty common workflow.

return the hash of a object (0 links, 0 data)

ipfs object blank # or 'ipfs object new'

Maybe ‘empty’ instead of ‘blank’ or ‘new’?

adds a link name -> hash

ipfs object patch add-link

How is this going to extend to extensible links? See ipfs/ipfs#36 and
#1172. How do we manipulate the data field? Also, drilling down into
a multi-node layout (e.g. chunked file or fanned-out directory) is
going to be tedious (but I'm fine with that for plumbing).

removes all links with name or hash or <int-offset>

ipfs object patch rm-link [ | | ]

where is the number in the link table.

( is only way to address uniquely. names and hashes may repeat)

How do you distinguish the int-offset ‘0’ from the name ‘0’?

@whyrusleeping
Copy link
Member Author

alright, so after implementing patch, doing what my save command here does is still very convoluted and difficult. Thoughts on actually merging this in one form or another?

@wking
Copy link
Contributor

wking commented May 29, 2015

On Thu, May 28, 2015 at 07:52:22PM -0700, Jeromy Johnson wrote:

alright, so after implementing patch, doing what my save command
here does is still very convoluted and difficult. Thoughts on
actually merging this in one form or another?

First round of porcelain: immediate bubbling. Pardon the shell, and
this isn't as robust as a Go implementation could be (e.g. it doesn't
handle /ipfs/Qm…/… paths):

#!/bin/sh

usage: replace.sh IPFS_PATH ACTION REPLACMENT

For example:

replace.sh QmZKetgwm4o3LhNaoLSHv32wBhTwj9FBwAdSchDMKyFQEx/foo/bar/baz add-link QmTz3oc4gdpRMKP2sdGUPZTAGRngqjsi99BPoztyP53JMM

will add a 'baz' entry to

QmZKetgwm4o3LhNaoLSHv32wBhTwj9FBwAdSchDMKyFQEx/foo/bar pointing at

QmTz3oc4gdpRMKP2sdGUPZTAGRngqjsi99BPoztyP53JMM, and then update

QmZKetgwm4o3LhNaoLSHv32wBhTwj9FBwAdSchDMKyFQEx/foo to point to that,

and then print out the new root hash.

set -e

dereference an IPFS path

dereference() {
local IPFS_PATH="$1"
local CURRENT=$(echo "${IPFS_PATH}" | cut -d / -f 1)
local NEXT=$(echo "${IPFS_PATH}" | cut -d / -f 2)
local TAIL=$(echo "${IPFS_PATH}" | cut -d / -f 3-)
while test -n "${NEXT}"
do
CURRENT=$(ipfs object get "${CURRENT}" |
jq --raw-output ".Links[] | select(.Name == "${NEXT}").Hash")
NEXT=$(echo "${TAIL}" | cut -d / -f 1)
TAIL=$(echo "${TAIL}" | cut -s -d / -f 2-)
done
echo "${CURRENT}"
}

replace_link() {
local ROOT="$1"
local NAME="$2"
local VALUE="$3"
local REMOVED=$(ipfs patch "${ROOT}" rm-link "${NAME}")
ipfs patch "${REMOVED}" add-link "${NAME}" "${VALUE}"
}

replace() {
local IPFS_PATH="$1" # QmZKetgwm4o3LhNaoLSHv32wBhTwj9FBwAdSchDMKyFQEx/foo/bar/baz
local ACTION="$2" # add-link
local REPLACEMENT="$3" # QmTz3oc4gdpRMKP2sdGUPZTAGRngqjsi99BPoztyP53JMM
local CURRENT=$(echo "${IPFS_PATH}" | cut -d / -f 1) # QmZ...Ex
local NEXT=$(echo "${IPFS_PATH}" | cut -d / -f 2) # foo
local TAIL=$(echo "${IPFS_PATH}" | cut -d / -f 3-) # bar/baz
if test -n "${TAIL}"
then
local CHILD=$(dereference "${CURRENT}/${NEXT}")"/${TAIL}" # {hash-for-foo}/bar/baz
REPLACEMENT=$(replace "${CHILD}" "${ACTION}" "${REPLACEMENT}")
replace_link "${CURRENT}" "${NEXT}" "${REPLACEMENT}"
else
case "${ACTION}" in
add-link)
replace_link "${CURRENT}" "${NEXT}" "${REPLACEMENT}"
;;
rm-link)
ipfs patch "${CURRENT}" "${ACTION}" "${NEXT}"
;;
*)
echo "unrecognized action: '${ACTION}'" >&2
exit 1
esac
fi
}

replace "$@"

Making ‘dereference’ a builtin command (‘ipfs object stat’) is #1082.
I'd be open to a add-or-replace-link action that matches my
‘replace_link’ function. I'd like to move the new command in #1299 to
‘ipfs object patch’ and expose a Go version of ‘replace’ as ‘ipfs
patch’. Once we have all of that, we can get back to talking about
saved sessions with commits ;).

@wking wking mentioned this pull request May 29, 2015
@whyrusleeping whyrusleeping added backlog and removed status/in-progress In progress labels May 31, 2015
@jbenet
Copy link
Member

jbenet commented Jun 2, 2015

alright, so after implementing patch, doing what my save command here does is still very convoluted and difficult. Thoughts on actually merging this in one form or another?

the patch porcelain on top of object patch?

@wking that's really good. 👍 on shell. we're on the git road ;) @chriscool would be proud. (i hope to be able to make prototyping command implementations in shell or other langs nicer before adding to api / core, etc).

@whyrusleeping said

Alternatively (or additionally) we could have an ipfs resolve command that outputs the hash that a path resolves to.

i like the

> ipfs resolve <path>
<hash>

idea (instead of dereference)

@whyrusleeping whyrusleeping mentioned this pull request Jun 2, 2015
58 tasks
@wking
Copy link
Contributor

wking commented Jun 2, 2015

On Mon, Jun 01, 2015 at 08:10:49PM -0700, Juan Batiz-Benet wrote:

Alternatively (or additionally) we could have an ipfs resolve
command that outputs the hash that a path resolves to.

i like the ipfs resolve <path> -> <hash> idea.

This is #1320 (and maybe also #1082), but I think we should think
through the intended command-line UI for resolution first (and that's
#1307). For example, ‘ipfs resolve …’ is a name that @jbenet doesn't
think should exist at all 1. But I'm not sure how that previous
comment squares with this new one ;).

@whyrusleeping
Copy link
Member Author

maybe we could rename this to something like ipfs unixfs insert or something. But we cannot acheive the same functionality through the patch command or similar. Imagine the fuse mountpoint is mounted, and two different programs are editing different files at different places in the tree. using the patch command, if one of those writes to the dag, it breaks the entire tree that the other one is on. This command goes through the ipnsfs interface and is threadsafe, and wont break the fuse interface.

@wking
Copy link
Contributor

wking commented Jun 4, 2015

On Thu, Jun 04, 2015 at 09:42:22AM -0700, Jeromy Johnson wrote:

Imagine the fuse mountpoint is mounted, and two different programs
are editing different files at different places in the tree. using
the patch command, if one of those writes to the dag, it breaks the
entire tree that the other one is on. This command goes through the
ipnsfs interface and is threadsafe, and wont break the fuse
interface.

Using the patch command shouldn't have any effect on the IPNS-linked
filesystem, since it's just creating new Merkle objects. The problem
seems to be the bug in publishing the new Merkle objects with ‘ipfs
name publish …’, and that's #1186.

@whyrusleeping
Copy link
Member Author

closing for now, will reopen later when i get to it

@Kubuxu Kubuxu deleted the feat/save branch February 27, 2017 20:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants