docs/code-flow : Add code flow documentation for add cmd. #5864

nitishm · 2018-12-20T16:56:34Z

License: MIT
Signed-off-by: Nitish Malhotra nitish.malhotra@gmail.com

schomatis · 2018-12-27T20:31:34Z

@nitishm hey, sorry, I totally forgot about this, I've been too focused on the MFS front when I should be prioritizing documents like this one that can have a much greater impact on our community.

schomatis

Very nice!

Let's add code references to a fixed go-ipfs version (and document it as such), e.g.,

https://github.com/ipfs/go-ipfs/tree/v0.4.18

We can use a concrete command example (this would help the reproducibility of this doc), e.g.,

# Convert a file to the IPFS format.
echo "Hello World!" > new-file
ipfs add new-file
# added QmfM2r8seH2GiRaC4esTjeraXEachRt8ZsSeGaWTPLyMoG new-file

# Add a file to the MFS.
NEW_FILE_HASH=$(ipfs add new-file -Q)
ipfs files cp /ipfs/$NEW_FILE_HASH /new-file
# Now we don't need the file hash anymore (which will even change in the
# future if we want to modify it), we can reference it through its MFS
# path: `/new-file`.

# Get information from the file in MFS.
ipfs files stat /new-file
# QmfM2r8seH2GiRaC4esTjeraXEachRt8ZsSeGaWTPLyMoG
# Size: 13
# CumulativeSize: 21
# ChildBlocks: 0
# Type: file
# TODO: Explain some of this fields.

# Retrieve the contents.
ipfs files read /new-file
# Hello World!

(ideally this example should be the same as the one in the Getting Started guide)

A next step that would be nice, taking advantage of your experience with Delve, could be to add an execution trace, showing the concrete values in certain function calls (we should maybe later create a different section about debugging go-ipfs), e.g.,

dlv version
Delve Debugger
Version: 1.0.0

dlv debug github.com/ipfs/go-ipfs/cmd/ipfs --build-flags "-tags noplugin" -- add new-file

# Inside `AddAllAndPin`
b coreunix/add.go:420

(dlv) c
> github.com/ipfs/go-ipfs/core/coreunix.(*Adder).AddAllAndPin() ./core/coreunix/add.go:420 (hits goroutine(30):1 total:1) (PC: 0x111fcd3)
   415:			// Iterate over each top-level file and add individually. Otherwise the
   416:			// single files.File f is treated as a directory, affecting hidden file
   417:			// semantics.
   418:			for {
   419:				f, err := file.NextFile()
=> 420:				if err == io.EOF {
   421:					// Finished the list of files.
   422:					break
   423:				} else if err != nil {
   424:					return nil, err
   425:				}
(dlv) p f
gx/ipfs/QmZMWMvWMVKCbHetJ4RgndbuEF1io2UpUxwQwtNjtYPzSC/go-ipfs-files.File(*gx/ipfs/QmZMWMvWMVKCbHetJ4RgndbuEF1io2UpUxwQwtNjtYPzSC/go-ipfs-files.ReaderFile) *{
	filename: "new-file",
	fullpath: "new-file",
	abspath: "/home/user/go/src/github.com/ipfs/go-ipfs/new-file",
	reader: io.ReadCloser(*os.File) *{
		file: *(*os.file)(0xc0000af800),},
	stat: os.FileInfo(*os.fileStat) *{
		name: "new-file",
		size: 13,
		mode: 436,
		modTime: (*time.Time)(0xc00020eab0),
		sys: (*syscall.Stat_t)(0xc00020eac8),},}


b Directory.AddChild
c
(dlv) stack
0  0x0000000000e37113 in gx/ipfs/QmUwXQs8aZ472DmXZ8uJNf7HJNKoMJQVa7RaCz7ujZ3ua9/go-mfs.(*Directory).AddChild
1  0x0000000000e3af4c in gx/ipfs/QmUwXQs8aZ472DmXZ8uJNf7HJNKoMJQVa7RaCz7ujZ3ua9/go-mfs.PutNode
2  0x000000000111f9dc in github.com/ipfs/go-ipfs/core/coreunix.(*Adder).addNode
3  0x0000000001120b48 in github.com/ipfs/go-ipfs/core/coreunix.(*Adder).addFile
4  0x000000000111ff39 in github.com/ipfs/go-ipfs/core/coreunix.(*Adder).AddAllAndPin
5  0x0000000001137a5e in github.com/ipfs/go-ipfs/core/coreapi.(*UnixfsAPI).Add

But check if this makes sense to you first, many of our functions are not very friendly and peeking inside may not reveal much useful information. Ideally, what I think would be really useful for a beginner would be (1) descriptive logs that would tell you what's going on inside without debugging and (2) external tools that would help shed some light on our internal structures, something like ipfs cid but for other components, or an extended version of ipfs object diff.

docs/add-code-flow.md

nitishm · 2019-01-06T18:49:28Z

I need to get to this soon.

Will try to wrap this up by the end of next week ! (sans the delve part, which might need me to do some investigation)

schomatis · 2019-01-06T18:54:20Z

Sounds good, let me know if you need any help.

nitishm · 2019-01-07T15:01:11Z

@schomatis I cleaned up the doc and formatted it in a better way.

Cant figure out what version of go-mfs to reference (kept it as master for now. Will change after your comment) ?

schomatis · 2019-01-07T15:53:16Z

Cant figure out what version of go-mfs to reference

That would be the version checked by the go-ipfs version we're documenting, in this case it would be

https://github.com/ipfs/go-ipfs/blob/aefc746f34e5ffdee5fba1915c6603b65a0ebf81/package.json#L533-L535

This is the Gx version that may not coincide with the actual go-mfs GitHub version (at some point publishing in Gx meant also tagging a release in the repo but I'm not sure now).

So, that version is not listed in the releases,

https://github.com/ipfs/go-mfs/releases

so basically what I do (there may be other ways, I hope) is to trace back the .gx/lastpubver which records the Gx version every time a gx publish is done, I go to the history,

https://github.com/ipfs/go-mfs/commits/master/.gx/lastpubver

and find the commit that corresponds to that Gx version published,

ipfs/go-mfs@8b5e8dd#diff-d95150d9db6fb31091770e6d6105b08d

where you can see the hash QmUwXQs8 that matches the one in package.json (the hash is always the ultimate source of truth in IPFS, and hence in Gx),

ipfs/go-mfs@8b5e8dd#diff-d95150d9db6fb31091770e6d6105b08dR1

Yes, this workflow definitely needs to be improved, I'm not sure why we stop tagging the Gx publishes in GitHub.

Stebalien · 2019-01-07T17:30:33Z

Yes, this workflow definitely needs to be improved, I'm not sure why we stop tagging the Gx publishes in GitHub.

whyrusleeping/gx#157

(all existing tags are created manually or with https://github.com/whyrusleeping/gx/blob/master/contrib/gx-retrotag.sh)

schomatis · 2019-01-07T17:38:24Z

Oh, too bad, I'll update the tags then.

License: MIT Signed-off-by: Nitish Malhotra <nitish.malhotra@gmail.com>

- Cleaned up some of the language. - Added versioned links to the code. - Provided examples License: MIT Signed-off-by: Nitish Malhotra <nitish.malhotra@gmail.com>

- go-mfs linked to v0.1.18 - go-unifx linked to v1.1.16 However, - go-merkledag has been linked to v1.1.13 which is not the right version. v1.1.15 has not been released and cannot be referenced License: MIT Signed-off-by: Nitish Malhotra <nitish.malhotra@gmail.com>

- Moved AddAllAndPin as a subsection of UnixfsAPI.Add() License: MIT Signed-off-by: Nitish Malhotra <nitish.malhotra@gmail.com>

nitishm · 2019-01-13T23:53:44Z

@schomatis Is there anything else you would like me to add to the docs ? The delve piece is still pending but I am not sure how do get enough information in that way. I think keeping it simple with the current format might be sufficient for now.

LMK what you think ?

schomatis

This is really nice, left some minor notes. I'd recommend to add:

A link to this document in https://github.com/ipfs/go-ipfs/#development to give more visibility to it.
A link here to https://github.com/ipfs/docs/issues/133 as a background read to get some idea of what terms like MFS and UnixFS mean (that is still a highly WIP document but it's the best we've got for the moment).

I'm still unsure what's the best format to present this kind of information but I'm in favor of getting this merged ASAP and iterate on it after we've got some feedback from the community.

docs/add-code-flow.md

- Cleaned up the document. - Added additional references to documents. License: MIT Signed-off-by: Nitish Malhotra <nitish.malhotra@gmail.com>

schomatis

Few nits and ready to go.

schomatis · 2019-01-15T15:49:35Z

docs/add-code-flow.md


            - **[UnixFS] [`(BasicDirectory).AddChild(ctx, name, ipld.Node)`](https://github.com/ipfs/go-unixfs/blob/v1.1.16/io/directory.go#L137)** - *Add child to `BasicDirectory`*

-                > IMPORTANT: It should be noted that the `BasicDirectory` struct of the `UnixFS` package, encasulates a node object of type `ProtoNode`, which is a different format from the `ipld.Node` we have been working on throughout this document.
+                > IMPORTANT: It should be noted that the `BasicDirectory` object uses the `ProtoNode` type object which is an implementation of the `ipld.Node` interface, seen and used throughout this document. Ideally the `ipld.Node` should always be used, unless we need access tp specific functions from `ProtoNode` (like `Copy()`) that are not available in the interface.


nit: typo, tp -> to.

schomatis · 2019-01-15T15:50:27Z

docs/add-code-flow.md

+
+# Also see
+1. https://github.com/ipfs/go-ipfs/#development


Sorry, I probably wasn't very clear, I meant for you to add a link in the https://github.com/ipfs/go-ipfs/#development section pointing here, so this document wouldn't get lost in the docs directory.

Oh you meant only in the issue conversation space and not the actual doc?! Oops!

In the actual entry-point README doc, but instead of pointing from here to there, point there to here.

Oh dang. I completely misunderstood. I didn't realize the link was the the top README.md doc. Alright I will go ahead and add it to the README.md

Oh, you're right, sorry, I didn't realize the link was copied without the README.md path 😬

schomatis · 2019-01-15T15:51:03Z

docs/add-code-flow.md

+# Also see
+1. https://github.com/ipfs/go-ipfs/#development
+2. [Concept document about files](https://github.com/ipfs/docs/issues/133)


We can probably move this to the introduction (before actually starting to talk about MFS/UnixFS).

- Fixed typo. - Moved concept document link to top of page. - Added link to this doc to README.md License: MIT Signed-off-by: Nitish Malhotra <nitish.malhotra@gmail.com>

schomatis

LGTM, thanks a lot for committing to this @nitishm, I know that documentation isn't the most exciting thing but I think that PRs like this can make a difference just as important as any other fix or feature 👍

schomatis · 2019-01-16T15:41:32Z

/cc @Stebalien

nitishm requested a review from Kubuxu as a code owner December 20, 2018 16:56

nitishm mentioned this pull request Dec 20, 2018

Document the code flow of adding a file using ipfs add <file> command. #5858

Open

schomatis requested review from schomatis and removed request for Kubuxu December 20, 2018 21:40

schomatis mentioned this pull request Dec 30, 2018

[WIP] doc: Files API #5052

Closed

schomatis reviewed Dec 30, 2018

View reviewed changes

docs/add-code-flow.md Outdated Show resolved Hide resolved

docs/add-code-flow.md Outdated Show resolved Hide resolved

docs/add-code-flow.md Outdated Show resolved Hide resolved

Stebalien added the need/author-input Needs input from the original author label Jan 4, 2019

schomatis added status/in-progress In progress and removed need/author-input Needs input from the original author labels Jan 7, 2019

schomatis assigned nitishm Jan 7, 2019

schomatis added this to the Files API Documentation milestone Jan 7, 2019

nmalhotra added 3 commits January 10, 2019 22:19

docs/code-flow : Add code flow documentation for add cmd.

75d5e40

License: MIT Signed-off-by: Nitish Malhotra <nitish.malhotra@gmail.com>

Cleanup and resolve docs

e229bc2

- Cleaned up some of the language. - Added versioned links to the code. - Provided examples License: MIT Signed-off-by: Nitish Malhotra <nitish.malhotra@gmail.com>

nitishm force-pushed the docs/mfs/add-code-flow branch from b21025d to 9e08723 Compare January 11, 2019 03:21

Fixed the indentation of the sections

d304a2f

- Moved AddAllAndPin as a subsection of UnixfsAPI.Add() License: MIT Signed-off-by: Nitish Malhotra <nitish.malhotra@gmail.com>

schomatis reviewed Jan 14, 2019

View reviewed changes

Document cleanup and additional references

7dbdb0e

- Cleaned up the document. - Added additional references to documents. License: MIT Signed-off-by: Nitish Malhotra <nitish.malhotra@gmail.com>

schomatis reviewed Jan 15, 2019

View reviewed changes

Added link to doc in README.md

269550b

- Fixed typo. - Moved concept document link to top of page. - Added link to this doc to README.md License: MIT Signed-off-by: Nitish Malhotra <nitish.malhotra@gmail.com>

nitishm force-pushed the docs/mfs/add-code-flow branch from ab44a46 to 269550b Compare January 16, 2019 02:09

schomatis approved these changes Jan 16, 2019

View reviewed changes

schomatis added topic/docs-ipfs Topic docs-ipfs RFM and removed status/in-progress In progress labels Jan 16, 2019

schomatis mentioned this pull request Jan 16, 2019

Cleaning up main REAMDE #5087

Open

Stebalien merged commit 74bf836 into ipfs:master Jan 21, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs/code-flow : Add code flow documentation for add cmd. #5864

docs/code-flow : Add code flow documentation for add cmd. #5864

nitishm commented Dec 20, 2018

schomatis commented Dec 27, 2018

schomatis left a comment

nitishm commented Jan 6, 2019

schomatis commented Jan 6, 2019

nitishm commented Jan 7, 2019

schomatis commented Jan 7, 2019

Stebalien commented Jan 7, 2019

schomatis commented Jan 7, 2019

nitishm commented Jan 13, 2019

schomatis left a comment

schomatis left a comment

schomatis Jan 15, 2019

schomatis Jan 15, 2019

nitishm Jan 15, 2019

schomatis Jan 15, 2019

nitishm Jan 15, 2019

schomatis Jan 15, 2019

schomatis Jan 15, 2019

schomatis left a comment

schomatis commented Jan 16, 2019


		# Also see
		1. https://github.com/ipfs/go-ipfs/#development

docs/code-flow : Add code flow documentation for add cmd. #5864

docs/code-flow : Add code flow documentation for add cmd. #5864

Conversation

nitishm commented Dec 20, 2018

schomatis commented Dec 27, 2018

schomatis left a comment

Choose a reason for hiding this comment

nitishm commented Jan 6, 2019

schomatis commented Jan 6, 2019

nitishm commented Jan 7, 2019

schomatis commented Jan 7, 2019

Stebalien commented Jan 7, 2019

schomatis commented Jan 7, 2019

nitishm commented Jan 13, 2019

schomatis left a comment

Choose a reason for hiding this comment

schomatis left a comment

Choose a reason for hiding this comment

schomatis Jan 15, 2019

Choose a reason for hiding this comment

schomatis Jan 15, 2019

Choose a reason for hiding this comment

nitishm Jan 15, 2019

Choose a reason for hiding this comment

schomatis Jan 15, 2019

Choose a reason for hiding this comment

nitishm Jan 15, 2019

Choose a reason for hiding this comment

schomatis Jan 15, 2019

Choose a reason for hiding this comment

schomatis Jan 15, 2019

Choose a reason for hiding this comment

schomatis left a comment

Choose a reason for hiding this comment

schomatis commented Jan 16, 2019