Add better support for Raw Nodes in MFS and elsewhere #3996

kevina · 2017-06-20T03:49:28Z

No description provided.

kevina · 2017-06-20T03:54:37Z

Towards #3989

kevina · 2017-06-20T06:45:57Z

Note this is a work in progress. In particular the Size() method in DagModifier needs to implemented to support raw nodes with MFS. I am trying to figure out if it is possible to get the equivalent of Filesize from the generic Node interface.

kevina · 2017-06-20T21:14:14Z

Okay this is no longer a work in progress. Fully generalize this to allow any type of node to be stored in the MFS tree should probably me another P.R.

Kubuxu · 2017-06-20T22:42:46Z

Could you add test cases from this PR: https://github.com/ipfs/go-ipfs/pull/3901/files

This makes sure that we test as much with RawNodes as we do test with normal ones.

whyrusleeping

Code LGTM, i would like to see another test in dagmodifier_test.go that tests dagmodifier.Size() on a raw node though

Stebalien

This mostly looks good to me (except as noted). However, I'm not very familiar with MFS.

As an aside, as you appear to be reasonably familiar with this subsystem, please consider documenting functions as you modify them (e..g, Size could mean "size of this node" but it looks like it means "file size" (number of user-specified bytes stored)).

Stebalien · 2017-06-20T23:15:51Z

unixfs/mod/dagmodifier.go

+	case *mdag.ProtoNode:
+		root = nd
+	case *mdag.RawNode:
+		// TODO: be able to append to rawnodes. Probably requires making this


In this case, you're appending a raw node to dm.curNode which may or may not be raw.

fair point. the comment and error are wrong then, maybe we can actually support this part fine now?

Wait, no; I'm totally wrong. All callers pass dm.currentNode. Now, as for why this function even takes a node...

@whyrusleeping I just removed your comment. If you wish to readd something please give me the correct text to use.

Stebalien · 2017-06-20T23:16:50Z

unixfs/mod/dagmodifier.go

-func (dm *DagModifier) appendData(node *mdag.ProtoNode, spl chunk.Splitter) (node.Node, error) {
+func (dm *DagModifier) appendData(nd node.Node, spl chunk.Splitter) (node.Node, error) {
+
+	var root *mdag.ProtoNode


Calling this root is misleading (it's the node to-be-appended).

This was not my change. @whyrusleeping care to comment

Yeah, root is probably not the best name here

I refactored that method to avoid the problem.

Stebalien · 2017-06-20T23:18:00Z

unixfs/mod/dagmodifier.go

 	dbp := &help.DagBuilderParams{
 		Dagserv:  dm.dagserv,
 		Maxlinks: help.DefaultLinksPerBlock,
 	}

-	return trickle.TrickleAppend(dm.ctx, node, dbp.New(spl))
+	return trickle.TrickleAppend(dm.ctx, root, dbp.New(spl))


The error in TrickleAppend should probably be changed to ErrNoRawYet (or dm.curNode should be checked here).

kevina · 2017-06-21T04:03:37Z

Could you add test cases from this PR: https://github.com/ipfs/go-ipfs/pull/3901/files

I thought the test I added where more comprehensive but I didn't see what you will trying to do. My added tests conflict with your but will make sure mine are just as comprehensive.

Kubuxu · 2017-06-21T22:24:18Z

test/sharness/t0250-files-api.sh

@@ -15,6 +15,7 @@ test_expect_success "can create some files for testing" '
 	FILE1=$(echo foo | ipfs add -q) &&
 	FILE2=$(echo bar | ipfs add -q) &&
 	FILE3=$(echo baz | ipfs add -q) &&
+        FILE9=$(echo zip | ipfs add -q --raw-leaves) &&


Indentation is different here than in the rest of the file

Yeah sorry about that. Tab vs space issue.

kevina · 2017-06-26T05:54:23Z

@Kubuxu I redid the tests so they are closer to what you had

Raw nodes are now working in the files API as far as I can tell.

kevina · 2017-06-26T05:55:21Z

@whyrusleeping

Code LGTM, i would like to see another test in dagmodifier_test.go that tests dagmodifier.Size() on a raw node though

It was non-obvious how to do this so I instead tested that the size was reported correctly in a sharness test.

whyrusleeping · 2017-06-26T17:22:03Z

@kevina The current dagmodifier tests use: https://github.com/ipfs/go-ipfs/blob/master/unixfs/test/utils.go#L45 which generates a random file of the given size, using a standard importer. You could make a new method there that creates a random file using the raw leaves importer option instead, and run some of the same tests on that node type in dagmodifier_test.go

kevina · 2017-06-26T21:48:11Z

@whyrusleeping most all of the tests in dagmodifier_test has to do with appending data, of which is not supported by raw leaves. It was non-obvious how to modify the test to support the creation of a node without appending. I can attempt (with some effort) to write a new set of tests for just raw leaves if you want.

As an aside, I am not sure of the benefit of supporting the full range of operations on raw leaves. Appending to a unixfs dir node is useful to add files. How is appending to a raw node useful? Is there even an API for appending to a file (I don't see it). Also when appending, what should we do when it becomes too large.

Kubuxu · 2017-06-26T22:18:13Z

@kevina yes there is an API (using ipfs files write), when it is too large it should probably be transformed into unixfs dag.

kevina · 2017-06-26T22:31:37Z

@Kubuxu, oh okay, append would be more complicated though and I think should be pushed to a separate p.r. With this one raw nodes are usable within the files api as long as you don't try to create one within the files API.

whyrusleeping · 2017-06-26T22:36:27Z

@kevina thats fair enough. I think we can live with that restriction for now.

kevina · 2017-06-26T22:42:44Z

@whyrusleeping are you okay with testing the size via the sharness tests only? The existing tests are designed for when the node is created within the API and I will have to write a new set of go tests to test that the raw node size is reported correctly, of which I am unsure how to do so I will have to spend some time figuring it out.

whyrusleeping · 2017-06-26T22:44:12Z

@kevina yeah, for now i guess testing through the api is good enough.

kevina · 2017-06-26T23:01:53Z

@whyrusleeping then I think this p.r. is ready

whyrusleeping · 2017-06-26T23:06:40Z

@kevina mind rebasing on top of latest master? Should make the tests green again

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

kevina added the status/in-progress In progress label Jun 20, 2017

kevina force-pushed the kevina/raw-nodes-fixes branch from b92e53b to 30c7730 Compare June 20, 2017 03:51

kevina changed the title ~~Add better support for Raw Nodes in MFS and elsewhere~~ WIP: Add better support for Raw Nodes in MFS and elsewhere Jun 20, 2017

whyrusleeping requested review from Stebalien and Kubuxu June 20, 2017 07:36

kevina force-pushed the kevina/raw-nodes-fixes branch from 30c7730 to 22e2f5a Compare June 20, 2017 21:05

kevina changed the title ~~WIP: Add better support for Raw Nodes in MFS and elsewhere~~ Add better support for Raw Nodes in MFS and elsewhere Jun 20, 2017

whyrusleeping requested changes Jun 20, 2017

View reviewed changes

Stebalien requested changes Jun 20, 2017

View reviewed changes

Stebalien approved these changes Jun 21, 2017

View reviewed changes

Kubuxu reviewed Jun 21, 2017

View reviewed changes

kevina force-pushed the kevina/raw-nodes-fixes branch from 6586737 to 0ab50f8 Compare June 23, 2017 23:25

kevina force-pushed the kevina/raw-nodes-fixes branch from 0ab50f8 to 7eef45c Compare June 26, 2017 06:41

Allow dagmodifier to be created (but not used) with raw nodes

c75ac6a

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

kevina added 3 commits June 26, 2017 19:12

Finish basic support for raw nodes in dag modifier.

4700154

License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

dagmodifer: refactor appendData method

3f6a7b7

License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

files api: redo tests

33f699f

License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

kevina force-pushed the kevina/raw-nodes-fixes branch from 7eef45c to 33f699f Compare June 26, 2017 23:17

kevina added the RFM label Jun 27, 2017

fix small typo

b3828f4

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

whyrusleeping approved these changes Jun 28, 2017

View reviewed changes

whyrusleeping merged commit 07162dd into master Jun 28, 2017

whyrusleeping removed the status/in-progress In progress label Jun 28, 2017

whyrusleeping deleted the kevina/raw-nodes-fixes branch June 28, 2017 02:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add better support for Raw Nodes in MFS and elsewhere #3996

Add better support for Raw Nodes in MFS and elsewhere #3996

kevina commented Jun 20, 2017

kevina commented Jun 20, 2017 •

edited

Loading

kevina commented Jun 20, 2017

kevina commented Jun 20, 2017

Kubuxu commented Jun 20, 2017

whyrusleeping left a comment

Stebalien left a comment

Stebalien Jun 20, 2017

whyrusleeping Jun 21, 2017

Stebalien Jun 21, 2017

kevina Jun 21, 2017

Stebalien Jun 20, 2017

kevina Jun 21, 2017

whyrusleeping Jun 21, 2017

kevina Jun 21, 2017

Stebalien Jun 20, 2017

kevina commented Jun 21, 2017

Kubuxu Jun 21, 2017

kevina Jun 22, 2017

kevina commented Jun 26, 2017

kevina commented Jun 26, 2017

whyrusleeping commented Jun 26, 2017

kevina commented Jun 26, 2017

Kubuxu commented Jun 26, 2017

kevina commented Jun 26, 2017

whyrusleeping commented Jun 26, 2017

kevina commented Jun 26, 2017

whyrusleeping commented Jun 26, 2017

kevina commented Jun 26, 2017

whyrusleeping commented Jun 26, 2017

Add better support for Raw Nodes in MFS and elsewhere #3996

Add better support for Raw Nodes in MFS and elsewhere #3996

Conversation

kevina commented Jun 20, 2017

kevina commented Jun 20, 2017 • edited Loading

kevina commented Jun 20, 2017

kevina commented Jun 20, 2017

Kubuxu commented Jun 20, 2017

whyrusleeping left a comment

Choose a reason for hiding this comment

Stebalien left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevina commented Jun 21, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevina commented Jun 26, 2017

kevina commented Jun 26, 2017

whyrusleeping commented Jun 26, 2017

kevina commented Jun 26, 2017

Kubuxu commented Jun 26, 2017

kevina commented Jun 26, 2017

whyrusleeping commented Jun 26, 2017

kevina commented Jun 26, 2017

whyrusleeping commented Jun 26, 2017

kevina commented Jun 26, 2017

whyrusleeping commented Jun 26, 2017

kevina commented Jun 20, 2017 •

edited

Loading