iroh add command

[faassen]

iroh-ctl add [--recursive | -r] [-q | --quiet] [--progress=true] [-w | --wrap-with-directory=true] [-n | --only-hash] [-H | --hidden] <path>

Upload to IPFS a file or directory given by path.

• -r or --recursive is required if path is a directory. If you omit it, the command fails with an error. This is to prevent you from uploading a directory by accident.
• --wrap=false can be supplied to prevent Iroh from wrapping what’s uploaded in a directory. This means that the filename information of the root path is lost.
• --progress=false suppresses the progress bar. It should default to that in non-interactive scenarios too.
• -q or --quiet makes all output go away, overriding any other options that control output.
• -n, --only-hash: only chunk and hash, do not write to disk.
• -H, --hidden: Include files that are hidden. Only takes effect on recursive add.

Differences with Kubo:

• ipfs add does not wrap what’s uploaded into a directory by default. This means that the filename of the root path or directory is not known to the network.
• ipfs add has a whole host of options to control the hashing algorithm and chunking.

Test cases

• Add a single file, success
• Add a directory, success
• Add a directory with subdirectories, success
• Add directory without -r, failure
• Adding a file or directory that does not exist, failure
• Turn off wrapping
• Only hash -- not sure how to test difference with other scenario, except to see whether an option is passed into unixfs. Not sure if supported yet
• Add a directory with hidden files without -H enabled
• Add a directory with hidden files and -H is enabled
• Add a file or directory that does not exist

trycmd tests are a bit limited here as we can't really see that the correct underlying APIs are called. Instead we will have to rely on API-level tests where hopefully the right factoring can make us detect that the correct underlying code is called.

Maybe later

• --column can be supplied multiple times to control the column names outputted. Pick from action, cid and path. If no --column entries exist, all columns are outputted.
• -C or --root-cid shows no columns or progress bar, only the final root CID by which this content can be outputted. This should override any --column specifications.
• adding from stdin

[faassen]

We want progress bars here first, not yet for iroh get. This needs to be multi progress bar, one per file, after an initial phase where it scans all the files you want to add.

[faassen]

I'll note the reason why I made -r not the default here briefly: it's to prevent people from accidentally uploading a whole directory, analogous to cp -r. I'm not hung up about this though.

In general I've tried to look at a lot of cp analogies but it breaks down because unfortunately we don't know whether we're getting a wrapped directory or not when we get.

[b5]

Reconciling your flags with mine:

• Instead of --no-wrap I'd prefer --wrap, which defaults to true. IMHO using negation in boolean flag labels is confusing: --wrap=false is clearer than --no-wrap
• ditto for --no-progress, I'd prefer --progress=false
• I don't think we need the --column flag quite yet, but I think it's well planned out and would not fight it in a future release.
• do we need the -q flag? I'd drop it unless bash magicians want to overrule me
• if we include --column, -C should override it.
• we should keep the -n, --only-hash bool - Only chunk and hash - do not write to disk. flag.
• we should keep the -H, --hidden bool - Include files that are hidden. Only takes effect on recursive add. flag.
• I think adding from stdin should be added in a later release

[b5]

in kubo <path> is variadic, allowing multiple paths to be ingested into the same DAG bound by a common root. We might need to support this in a later release, but we do need to do it at some point. I'm fine if v0.1.0 only takes a single path

[b5]

Full help text, also matched to the CLI spike branch:

$ iroh add --help
iroh-add
Add a file to iroh & make it available on IPFS

USAGE:
    iroh add [OPTIONS] <PATH>

ARGS:
    <PATH>    The path to a file or directory to be added

OPTIONS:
    -C, --root-cid               Output only the final Content Identifier (CID)
    -h, --help                   Print help information
    -H, --hidden                 Include files that are hidden. Only takes effect on recursive add.
    -n, --only-hash              Only chunk and hash. Do not write to disk.
    -p, --progress               Stream progress data. Default: true
    -r, --recursive              Add directory paths recursively. Default: false
    -w, --wrap-with-directory    Wrap files with a directory object. Default: true


NOTE: IROH CURRENTLY PROVIDES NO WAY TO REMOVE CONTENT ONCE ADDED. This will be
addressed in a future release.

Add copies the file or directory specified by of <path> into the iroh store,
splitting the input file into a tree of immutable blocks. Each block is labeled
by the hash of its content. The final output of the add command is the hash of
the root of the tree, which contains references to all other blocks:

  > iroh add example.jpg
  added [HASH] example.jpg
  added [ROOT_CID]

[ROOT_CID] is a Content IDentifier (or CID), which is a hash plus some extra
metadata. The opposite of the add command is the get command, which accepts a
CID and turns it back into files or directories:

  > iroh get [ROOT_CID]
  [ROOT_CID] written to working directory

The stored result of add is a 'MerkleDAG'. Merkle proofs (hashes) are a fast
method of proving and checking data inclusion, and the tree formed by chunking
the input into blocks is always a directed acyclic graph (DAG). These MerkleDAGs
can be provably checked for tamper resistance by anyone who fetches all blocks
in the tree, which means MerkleDAGs can be provided by anyone, without concern
for tampering.

Once content is added to iroh, it can be provided & rehosted by any other IPFS
node with iroh start:

  > iroh start

By default all content added to iroh is made available to the network, and the
default network is the public IPFS network. We can prove this by creating an
empty iroh instance. Run this in a separate terminal with iroh start running:

  > IROH_PATH=iroh_temp && iroh get [ROOT_CID]

We can use an HTTPS gateway hosted at https://gateway.lol to fetch the content
from the node running with iroh start:

  > curl https://gateway.lol/ipfs/[ROOT_CID]/example.jpg

In this case we're trusting whoever runs gateway.lol to verify the merkle proof
for us, because the last mile of our request is over HTTPS, whereas 'iroh get'
always performs this check locally.

The wrap option, '-w', wraps the file (or files, if using the recursive option)
in a directory and is on by default. This directory contains only the files
which have been added so the input retains its filename. set -w=false for
'raw' adds that lose their input name:

    > iroh add example.jpg
    added [CID] example.jpg
    added [ROOT_CID]
    > iroh add example.jpg -w=false
    added [ROOT_CID] example.jpg

Note that in both cases the CID of example.jpg is the same. Adding the same
file/directory with the same flags is deterministic, which means the same input
paired with the same configuration flags will give the same root hash.

[faassen]

I don't think we need the --column flag quite yet, but I think it's well planned out and would not fight it in a future release.

Given that we have a progress bar based user interface this is likely less relevant. I think we may be better off with a --format=json that just dumps all columns for the use case where people need data.

[faassen]

The whole

    added [CID] example.jpg
    added [ROOT_CID]

bit doesn't seem like we'll implement it in this form, given that we anticipate having a single progress bar with a status line that indicates the current file. #183

[faassen]

I imagine --only-hash does not actually upload or announce the chunks anymore and is really a kind of test run? Is that correct? "Does not write to disk" is confusing me -- why would iroh add ever write to disk? Does this mean do not write to the store"?

[faassen]

I'm not sure how to implement short forms of -w and -p. These flags are on by default, and in this they're not really flags at all, but values. Is -p=false a thing people do in CLIs? See #156 for a discussion about the technical limitations.

[b5]

I imagine --only-hash does not actually upload or announce the chunks anymore and is really a kind of test run? Is that correct? "Does not write to disk" is confusing me -- why would iroh add ever write to disk? Does this mean do not write to the store"?

yes --only-hash should not write to the store, not announce, and drop all chunks created. It's basically used as a "what would iroh call this" function. Let's punt the --only-hash flag to a later release. Based on your confusion I think it merits more thought

I'm not sure how to implement short forms of -w and -p. These flags are on by default, and in this they're not really flags at all, but values. Is -p=false a thing people do in CLIs? See #156 for a discussion about the technical limitations.

Don't implement short forms in this case. -p=false destroys the legibility argument entirely. Users should have to explicitly supply --wrap=false.

[faassen]

If I understand it correctly then wee could call --only-hash --dry-run as that's what tools often name it. Once we implement the capability.