Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix progress bar for "get" command #1279

Closed
wants to merge 3 commits into from
Closed

Conversation

rht
Copy link
Contributor

@rht rht commented May 22, 2015

#736

Should look like this

go-ipfs (cleanup-progressbar) x ipfs get QmQv4YQNmRPuTTHs4AgBhKEFDdN7eQYeTbSmr8JVWVfury/sintel.mp4    ~/code/gocode/src/github.com/ipfs/go-ipfs
Saving file(s) to sintel.mp4
28.00 MB / 123.28 MB [=========================>----------------------------------------------------------------------------------------] 22.71 % 7m18s

@whyrusleeping I use NodeState instead of pbdata. Is this more efficient?

return nil, 0, err
}

length := uint64(ns.CumulativeSize)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think the cumulative size will be the total ipfs size, no the total unixfs size. (meaning, the ipfs size has some overhead). Maybe this is fine. (and maybe this is why the add / cat progressbar sometimes goes past 100%...)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

afaik add progressbar doesn't use ipfs size, so it's fine. I don't know which one dagreader uses.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh it is more correct to use ipfs size? because of the >100%?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, the ipfs size is the real amount of data moved. the unixfs size may be smaller, but could even be larger (say if i have 1000 links to the same, huge object). whatever we pick we should use it correctly (i.e. the >100% i think is because they're perhaps mixed)

@jbenet
Copy link
Member

jbenet commented May 22, 2015

I'm very excited to have this merged-- thank you for picking this up @rht

@whyrusleeping
Copy link
Member

I think using the CumulativeSize from the node stat should work well. its a better representation of the data involved.

@jbenet
Copy link
Member

jbenet commented May 30, 2015

expecting success: 
      mkdir -p dir &&
      touch dir/a &&
      mkdir -p dir/b &&
      echo "Hello, Worlds!" >dir/b/c &&
      HASH2=`ipfs add -r -q dir | tail -n 1` &&
      ipfs get "$HASH2" >actual

Error: unexpected EOF
not ok 36 - ipfs get succeeds (directory)
#   
#         mkdir -p dir &&
#         touch dir/a &&
#         mkdir -p dir/b &&
#         echo "Hello, Worlds!" >dir/b/c &&
#         HASH2=`ipfs add -r -q dir | tail -n 1` &&
#         ipfs get "$HASH2" >actual
#       
expecting success: 
      printf "%s\n\n" "Saving file(s) to $HASH2" >expected &&
      test_cmp expected actual

> diff -u expected actual
--- expected    2015-05-22 09:59:00.930957429 +0000
+++ actual  2015-05-22 09:59:00.920958007 +0000
@@ -1,2 +1 @@
 Saving file(s) to QmbiZReo9jpGMrVtNKNuNpLA5cLX3vj5F24wp3xemdzCAn
-
not ok 37 - ipfs get output looks good (directory)
#   
#         printf "%s
#   
#   " "Saving file(s) to $HASH2" >expected &&
#         test_cmp expected actual
#       
expecting success: 
      test_cmp dir/a "$HASH2"/a &&
      test_cmp dir/b/c "$HASH2"/b/c &&
      rm -r "$HASH2"

diff: QmbiZReo9jpGMrVtNKNuNpLA5cLX3vj5F24wp3xemdzCAn/a: No such file or directory
> diff -u dir/a QmbiZReo9jpGMrVtNKNuNpLA5cLX3vj5F24wp3xemdzCAn/a
diff: QmbiZReo9jpGMrVtNKNuNpLA5cLX3vj5F24wp3xemdzCAn/a: No such file or directory
not ok 38 - ipfs get output is valid (directory)
#   
#         test_cmp dir/a "$HASH2"/a &&
#         test_cmp dir/b/c "$HASH2"/b/c &&
#         rm -r "$HASH2"
#       
expecting success: 
      ipfs get "$HASH2" -a -C >actual

0 B / 166 B [----------------------------------------------------------] 0.00 % Error: unexpected EOF
not ok 39 - ipfs get -a -C succeeds (directory)
#   
#         ipfs get "$HASH2" -a -C >actual
#       

@jbenet
Copy link
Member

jbenet commented May 30, 2015

@rht sorry for delay here o/

@rht rht force-pushed the cleanup-progressbar branch 2 times, most recently from 2d034f8 to 1170fd2 Compare May 30, 2015 06:24
@rht
Copy link
Contributor Author

rht commented May 30, 2015

'echo "!"' doesn't escape the '!'
edit: nope this is orthogonal to the issue.

@rht rht force-pushed the cleanup-progressbar branch 3 times, most recently from 5e269f9 to bf7d735 Compare May 30, 2015 17:55
@rht
Copy link
Contributor Author

rht commented May 30, 2015

Sorry will fix this as soon as I can.

@whyrusleeping whyrusleeping mentioned this pull request Jun 2, 2015
58 tasks
@jbenet
Copy link
Member

jbenet commented Jun 3, 2015

@rht any luck with this? would love to have it in-- get is painful without it

@rht rht force-pushed the cleanup-progressbar branch 14 times, most recently from 18e8a83 to 53f80c3 Compare June 4, 2015 05:13
@rht rht force-pushed the cleanup-progressbar branch 4 times, most recently from 703d6de to 97ed6e0 Compare June 15, 2015 23:28
@whyrusleeping
Copy link
Member

Is any help from me needed here?

@rht
Copy link
Contributor Author

rht commented Jun 17, 2015

Can't figure out the source of unexpected EOF here: https://travis-ci.org/ipfs/go-ipfs/jobs/66950969#L2886, https://travis-ci.org/ipfs/go-ipfs/jobs/66950969#L2918, https://travis-ci.org/ipfs/go-ipfs/jobs/66950969#L2918. Can't reproduce locally (sharness test passes just fine)

@jbenet
Copy link
Member

jbenet commented Jun 30, 2015

@rht an update here? what's needed?

@rht rht force-pushed the cleanup-progressbar branch 3 times, most recently from f50b7d4 to 6086258 Compare July 4, 2015 14:29
@rht rht force-pushed the cleanup-progressbar branch 2 times, most recently from 30b963a to 65963e0 Compare August 8, 2015 08:31
License: MIT
Signed-off-by: rht <rhtbot@gmail.com>
rht referenced this pull request Aug 9, 2015
up until now there has been a very annoying bug with get, we would
get halting behavior. I'm not 100% sure this commit fixes it,
but it should. It certainly fixes others found in the process of
digging into the get / tar extractor code. (wish we could repro
the bug reliably enough to make a test case).

This is a much cleaner tar writer. the ad-hoc, error-prone synch
for the tar reader is gone (with i believe was incorrect). it is
replaced with a simple pipe and bufio. The tar logic is now in
tar.Writer, which writes unixfs dag nodes into a tar archive (no
need for synch here). And get's reader is constructed with DagArchive
which sets up the pipe + bufio.

NOTE: this commit also changes this behavior of `get`:
When retrieving a single file, if the file exists, get would fail.
this emulated the behavior of wget by default, which (without opts)
does not overwrite if the file is there. This change makes get
fail if the file is available locally. This seems more intuitive to
me as expected from a unix tool-- though perhaps it should be
discussed more before adopting.

Everything seems to work fine, and i have not been able to reproduce
the get halt bug.

License: MIT
Signed-off-by: Juan Batiz-Benet <juan@benet.ai>
@jbenet
Copy link
Member

jbenet commented Aug 9, 2015

@rht so this is fixed, then?

@rht
Copy link
Contributor Author

rht commented Aug 9, 2015

Yes it is.

My only concern here is when calculating the tar size, this is needed:

if err := proto.Unmarshal(nd.Data, pb); err != nil {
        return 0, err
}

Does this download the blocks underneath?

@jbenet
Copy link
Member

jbenet commented Aug 9, 2015

@rht the proto.Unmarshal does not download blocks underneath, it merely decodes the encoded data. (the ipfs ls command downloads them to check the types, so it can draw a / on dirs/)

@rht
Copy link
Contributor Author

rht commented Aug 9, 2015

  • get .tar progressbar
  • get .tar.gz progressbar
  • get .gz progressbar

ok it shouldn't be confusing to use uncompressed size in .tar.gz and .gz (but the progressbar reader has to be slipped in between tarwriter and gzwriter)?

rht added 2 commits August 9, 2015 23:24
License: MIT
Signed-off-by: rht <rhtbot@gmail.com>
License: MIT
Signed-off-by: rht <rhtbot@gmail.com>
@rht
Copy link
Contributor Author

rht commented Aug 9, 2015

@jbenet @whyrusleeping it appears that thirdparty/tar/extractor.go can be removed completely.
Because t is used only when the output is not compressed. i.e.
ipfs getdag | dag2tar | 3rdparty_untar -o unixfs

Why not just
ipfs getdag -o unixfs.

@whyrusleeping
Copy link
Member

@rht kill it!

@jbenet
Copy link
Member

jbenet commented Aug 9, 2015

I believe it is also used to download tarballs via the API. That way we can curl > .tar


Sent from Mailbox

On Sun, Aug 9, 2015 at 8:24 PM, Jeromy Johnson notifications@github.com
wrote:

@rht kill it!

Reply to this email directly or view it on GitHub:
#1279 (comment)

@rht
Copy link
Contributor Author

rht commented Aug 9, 2015

I believe it is also used to download tarballs via the API. That way we can curl > .tar

Nope, if you grep 'github.com/ipfs/go-ipfs/thirdparty/tar', it is only used by ipfs get to extract intermediary tar-ed dag (not tar-ed file that was put into dag).
For curl pattern, use ipfs cat <hash> | tar xf (assuming the knowledge that the hash contains a tar).

@whyrusleeping
Copy link
Member

@rht i was using it in ipfs shell until today when i refactored and pulled the package out here: https://github.com/whyrusleeping/tar-utils

@rht
Copy link
Contributor Author

rht commented Aug 10, 2015

@whyrusleeping it is used because the uncompressed output is still tar-formatted? Though if #1558 happens, func (s *Shell) Get(hash, outdir string) error will become more complex for recursive get.

@rht
Copy link
Contributor Author

rht commented Aug 10, 2015

Ok this is not more complex. tar-utils simply need to be replaced with a generic dag walker. This can be both imported here and used in ipfs get.

@rht rht closed this Oct 3, 2015
@jbenet jbenet removed the backlog label Oct 3, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants