add a --with-local option to ipfs files stat #4638

MichaelMure · 2018-02-01T12:08:10Z

Ok, new attempt after #4541 and #4596.

This PR add a --with-local option to ipfs files stat that compute and display how much of a dag is local.

I had a small problem with a the functions defined in env.go that are not reachable from a subdir of core/commands so I copied the file for now. I'm not sure how you want to solve that.

On a side note, I would be very pleased if any version can make it to the upcoming RC and version.

MichaelMure · 2018-02-01T13:57:59Z

@whyrusleeping @Stebalien @kevina please have a look.

kevina · 2018-02-01T20:13:32Z

I would also like to have @keks look over this if he is available as you are switching over the to new commands library.

kevina

Left one comment.

In addition the file core/commands/files/env.go is a copy of core/commands/env.go please remove it.

kevina · 2018-02-01T20:00:54Z

core/commands/files/files.go

+	CumulativeSize uint64
+	Blocks         int
+	Type           string
+	WithLocality   bool


Please add `json:",omitempty"` after WithLocality so that the API output does not change if the --with-local option is not used.

Someone to weight here ? For me it makes more sense like this. There is no compatibility problem, it's just an extra field. The problem is If WithLocality is set to false, the field is not present in the api anwser so you need to test if the field is present and if it's set to true. Not the end of the world, but ...

@MichaelMure No, the same struct would be used on both sides. theres no need to check if its set, we just don't want the API to change where its not wanted. @kevina is right here.

kevina · 2018-02-01T21:05:34Z

I had a small problem with a the functions defined in env.go that are not reachable from a subdir of core/commands so I copied the file for now. I'm not sure how you want to solve that.

Part of the problem is the files.go is in its own directory but not a proper package. Since this is just a single file, the easiest solution is to just move it. Commit b573768 (branch kevina/statlocal-fix1) does this. Feel free to cherry pick that change.

The other solution is to properly make files.go into its own package. I tried that in 554752c (branch kevina/statlocal-fix2) but ran into a import cycle:

import cycle not allowed
package github.com/ipfs/go-ipfs/cmd/ipfs
        imports github.com/ipfs/go-ipfs/core/commands
        imports github.com/ipfs/go-ipfs/core/commands/files
        imports github.com/ipfs/go-ipfs/core/commands
cmd/ipfs/Rules.mk:22: recipe for target 'cmd/ipfs/ipfs' failed

With a little more effort this could be properly fixed, but I think its best just to move files.go out of it own directory.

whyrusleeping · 2018-02-02T19:01:56Z

core/commands/files.go

-		"rm":    FilesRmCmd,
-		"flush": FilesFlushCmd,
-		"chcid": FilesChcidCmd,
+		"read":  lgc.NewCommand(filesReadCmd),


Alternatively, You could just use the OldSubcommands map instead of calling legacy.NewCommand on all of these. I'm not that picky though.

whyrusleeping · 2018-02-02T19:04:24Z

core/commands/files.go

-		switch fsn.Type() {
-		case mfs.TDir:
+		switch d.GetType() {
+		case ft.TDirectory, ft.THAMTShard:


why make this change?

d.GetType() can be of these types so if HAMTSharding was enabled we could potentially get an unrecognized node type error here, ya?

@whyrusleeping because we want (or at least I do) support /ipfs/Qmwhatever. getNodeFromPath will return an ipld.Node instead of a mfs.FSNode.

@frrist indeed. I got that from https://github.com/ipfs/go-ipfs/blob/master/mfs/system.go#L92 where there is the same problem. Would that be a file, directory or else ?

I think they must either be a file or a directory,

Files is an API for manipulating IPFS objects as if they were a unix filesystem. - file.go

The current definition looks good to me, what do ya'll think?

Ideally, MFS would handle /ipfs/ as a "mounted" subdirectory but we can fix that later.

whyrusleeping · 2018-02-02T19:15:54Z

I would really prefer that we not move files.go back up into the "all the commands" directory. But it does make it pretty obnoxious to keep it down there...

kevina · 2018-02-02T20:04:31Z

@whyrusleeping

I would really prefer that we not move files.go back up into the "all the commands" directory. But it does make it pretty obnoxious to keep it down there...

To fix the import cycle we will need to move env.go into its own package also. I updated the kevina/statlocal-fix2 branch to do this in ae7324a.

keks · 2018-02-02T20:06:56Z

FWIW I never understood why files, object etc. had their own subfolders. Why shouldn't they be in the main command folder?

whyrusleeping · 2018-02-02T20:10:50Z

Yeah, it was really just because I wanted to try and break things up into smaller bits. I'm okay with having them all in the one folder now.

kevina · 2018-02-02T20:20:34Z

I'm okay either way. One reason files.go object.go and dht.go are in there own package is because they have there own prefix in the logger. When I moved files.go from its own package I needed to rename log to flog.

The disadvantage of having them in there own package is that common code will also need to be in its own package as I did in ae7324a, in case you missed it.

whyrusleeping · 2018-02-03T23:34:30Z

core/commands/files.go

-			}
+		withLocal, _ := req.Options["with-local"].(bool)
+		if !withLocal {
+			cmds.EmitOnce(res, o)


you should just be able to use res.Emit(o)

cc @keks

whyrusleeping · 2018-02-03T23:35:56Z

Alright, this LGTM. Will need a rebase.

Can I get a +1 from @Stebalien and @kevina ?

kevina · 2018-02-04T00:17:24Z

@whyrusleeping @MichaelMure because of the file move you need to be careful when reserving the conflict. Now that I worked out how to fix files.go so it can be in its own package, that might be best.

Note that I can take over the p.r. if it would help.

License: MIT Signed-off-by: Michael Muré <batolettre@gmail.com>

It is a single file so putting it in its own package is a bit of an overkill. License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT Signed-off-by: Michael Muré <batolettre@gmail.com>

MichaelMure · 2018-02-04T01:01:55Z

@whyrusleeping rebase is done.
@kevina please take over if you want to make further changes. I can't push to ipfs/go-ipfs so I added you as a collaborator on my own repo if you want to stay in the same place.

kevina · 2018-02-04T01:44:38Z

I double checked the rebase and it looks good, the conflict was due to a dependency update which I made sure was correct.

@MichaelMure you should not need to give me access as I can push directly to your branch since you made a p.r. from it.

@whyrusleeping as I said before now that I think about it I am not sure moving files.go out of its directory is the best way and I have figured out to resolve the dependency cycle in ae7324a. object.go and dag.go are also in there own directory. What ever we do we should make things uniform.

[Truth be told I am okay either way, but it seams my comments are being lost in the noise and I am not sure anyone saw by commit that fixed the cycle. I apologize if I making things more complicated than need be.]

whyrusleeping

Alright, this LGTM.

@MichaelMure You've tested this with your application right? I wouldnt want to merge this is only to not have it work out right for you.

MichaelMure · 2018-02-05T21:43:29Z

@whyrusleeping it does works. Good thing is that I didn't even need to change js-ipfs-api.

Stebalien

LGTM. Not so happy with the naming (--with-local, WithLocality, etc.) but I can't come up with better names so... .

Stebalien · 2018-02-08T02:35:14Z

core/commands/files.go

-		switch fsn.Type() {
-		case mfs.TDir:
+		switch d.GetType() {
+		case ft.TDirectory, ft.THAMTShard:


Ideally, MFS would handle /ipfs/ as a "mounted" subdirectory but we can fix that later.

whyrusleeping · 2018-02-08T22:21:43Z

Thanks for putting up with our indecision @MichaelMure :)

MichaelMure · 2018-02-08T22:42:43Z

MichaelMure force-pushed the statlocal branch from 0361d05 to 7538f25 Compare February 1, 2018 12:24

kevina requested review from kevina, keks, Stebalien and whyrusleeping February 1, 2018 20:12

kevina suggested changes Feb 1, 2018

View reviewed changes

whyrusleeping reviewed Feb 2, 2018

View reviewed changes

whyrusleeping reviewed Feb 3, 2018

View reviewed changes

MichaelMure and others added 7 commits February 4, 2018 01:51

ipfs files: use go-ipfs-cmds

53c42fc

License: MIT Signed-off-by: Michael Muré <batolettre@gmail.com>

ipfs files stat: use go-ipfs-cmds

d716a93

License: MIT Signed-off-by: Michael Muré <batolettre@gmail.com>

add a --with-local option to ipfs files stat

69b8383

License: MIT Signed-off-by: Michael Muré <batolettre@gmail.com>

add tests for ipfs files stat --with-local

8fd6fdb

License: MIT Signed-off-by: Michael Muré <batolettre@gmail.com>

cleaning

cac995e

License: MIT Signed-off-by: Michael Muré <batolettre@gmail.com>

Move files.go out of its own directory.

eeb15db

It is a single file so putting it in its own package is a bit of an overkill. License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

fix linting

189c905

License: MIT Signed-off-by: Michael Muré <batolettre@gmail.com>

MichaelMure force-pushed the statlocal branch from 3faf668 to e0babef Compare February 4, 2018 00:55

address code review

30d4eaa

License: MIT Signed-off-by: Michael Muré <batolettre@gmail.com>

MichaelMure force-pushed the statlocal branch from e0babef to 30d4eaa Compare February 4, 2018 00:58

whyrusleeping approved these changes Feb 5, 2018

View reviewed changes

Stebalien approved these changes Feb 8, 2018

View reviewed changes

whyrusleeping merged commit 25eeb1e into ipfs:master Feb 8, 2018

MichaelMure mentioned this pull request Feb 19, 2018

add support for ipfs files stat --with-local ipfs-inactive/js-ipfs-http-client#695

Merged

MichaelMure mentioned this pull request Feb 28, 2018

complete files.stat with the 'with-local' option ipfs-inactive/interface-js-ipfs-core#227

Merged

This was referenced Dec 5, 2018

Implement the command "ipfs repo has" #4541

Closed

Add a 'ipfs files tree' command #4596

Closed

Stebalien mentioned this pull request Dec 9, 2019

feat: files (mfs) api ipfs/interface-go-ipfs-core#54

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add a --with-local option to ipfs files stat #4638

add a --with-local option to ipfs files stat #4638

MichaelMure commented Feb 1, 2018

MichaelMure commented Feb 1, 2018

kevina commented Feb 1, 2018 •

edited

Loading

kevina left a comment

kevina Feb 1, 2018 •

edited

Loading

MichaelMure Feb 2, 2018

whyrusleeping Feb 2, 2018

MichaelMure Feb 2, 2018

kevina commented Feb 1, 2018 •

edited

Loading

whyrusleeping Feb 2, 2018

whyrusleeping Feb 2, 2018

frrist Feb 2, 2018

MichaelMure Feb 2, 2018

MichaelMure Feb 2, 2018

frrist Feb 2, 2018

Stebalien Feb 8, 2018

whyrusleeping commented Feb 2, 2018

kevina commented Feb 2, 2018

keks commented Feb 2, 2018

whyrusleeping commented Feb 2, 2018

kevina commented Feb 2, 2018

whyrusleeping Feb 3, 2018

whyrusleeping commented Feb 3, 2018

kevina commented Feb 4, 2018

MichaelMure commented Feb 4, 2018

kevina commented Feb 4, 2018

whyrusleeping left a comment

MichaelMure commented Feb 5, 2018

Stebalien left a comment

Stebalien Feb 8, 2018

whyrusleeping commented Feb 8, 2018

MichaelMure commented Feb 8, 2018

add a --with-local option to ipfs files stat #4638

add a --with-local option to ipfs files stat #4638

Conversation

MichaelMure commented Feb 1, 2018

MichaelMure commented Feb 1, 2018

kevina commented Feb 1, 2018 • edited Loading

kevina left a comment

Choose a reason for hiding this comment

kevina Feb 1, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevina commented Feb 1, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

whyrusleeping commented Feb 2, 2018

kevina commented Feb 2, 2018

keks commented Feb 2, 2018

whyrusleeping commented Feb 2, 2018

kevina commented Feb 2, 2018

Choose a reason for hiding this comment

whyrusleeping commented Feb 3, 2018

kevina commented Feb 4, 2018

MichaelMure commented Feb 4, 2018

kevina commented Feb 4, 2018

whyrusleeping left a comment

Choose a reason for hiding this comment

MichaelMure commented Feb 5, 2018

Stebalien left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

whyrusleeping commented Feb 8, 2018

MichaelMure commented Feb 8, 2018

kevina commented Feb 1, 2018 •

edited

Loading

kevina Feb 1, 2018 •

edited

Loading

kevina commented Feb 1, 2018 •

edited

Loading