Add "ipfs block rm" command. #2962

kevina · 2016-07-10T21:44:04Z

This is a basic implementation for #2914.

It accepts multiple blocks to remove, however that does not provide any performance benifit just yet as it calls pins.IsPinned() for each block, which traverses every recursive pin to check for indirect pins. This operation costs is O(P*N) where P is the number of pinned blocks and N is the number of blocks to be removed. With a little effort this can be brought down to O(P) by checking for multiple pins at once.

Since the pin check is expensive it provides an option to ignore pins. Removing a pinned block will break any command that checks for pins. Right now if this happens recovering from the situation is very difficult. At very least a command should be provided to list any broken or missing pins. An additional optional command can be provided to attempt to repair recursive pins.

TODO:

Provide command to check for multiple pins at once

No longer relevant:

[] Provide a ipfs pin verify command to list broken or missing pins

Optional and No longer relevant

[] Provide a command to repair pins (I already wrote similar code in [WIP] Filestore Implementation #2634)

RichardLitt · 2016-07-11T10:35:38Z

Nice! Subscribing, for the http-api-spec.

whyrusleeping · 2016-07-13T08:46:52Z

core/commands/block.go

+		cmds.BoolOption("ignore-pins", "Ignore pins.").Default(false),
+	},
+	Run: func(req cmds.Request, res cmds.Response) {
+		ignorePins, _, err := req.Option("ignore-pins").Bool()


having this option makes me very nervous. Things break rather spectaculary when we have missing blocks that are pinned

I Know. But I see is as necessary, at least until we can get pinning operations to be O(1). Please also see my initial description on this pr and #2963.

whyrusleeping · 2016-08-04T14:09:49Z

I'm still not comfortable having the option to ignore pins here, letting a single command have the power to corrupt your repo isnt a very good idea.

kevina · 2016-08-10T21:00:35Z

@whyrusleeping please have another look. I send the results as a channel and use PostRun now to process the results. I also avoid aborting on a non-fatal error and instead just report that the block can't be deleted and continue on to the next block.

I am okay with removing the option to ignore pins, but I want to fix the pin checks so it is O(p) instead of O(p*n) where p is the number of pinned blocks and n is the number of blocks to be removed.

whyrusleeping · 2016-08-10T23:15:42Z

Yeah, if we're removing that option then that all looks pretty good.

kevina · 2016-08-15T03:59:40Z

@whyrusleeping okay, for now I removed the option to ignore pins

If you are happy with this fell free to merge.

License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

Provide a new method, Pinner.CheckIfPinned(), which will check if any of the arguments are pinned. Previously IsPinned would need to be called once for each block. The new method will speed up the checking of multiple pinned blocks from O(p*n) to O(p) (where p is the number of pinned blocks and n is the number of blocks to be check) Use the new method in "block rm". License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

daviddias · 2016-08-16T11:13:51Z

@whyrusleeping do you need anything else for this PR? Looking forward to have #2914 so that ipfs-inactive/js-ipfs-http-client#305 can be finished with regards to the Block API :)

whyrusleeping · 2016-08-16T16:14:32Z

This looks pretty good to me, it sucks that we have to be so hacky with the commands lib PostRun stuff. Maybe we can figure out a better way to restructure that code in the future.

Only thing i'd want to see added here is a test of removing multiple blocks, where a subset of them are pinned.

License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

kevina · 2016-08-16T23:52:18Z

@whyrusleeping okay I added a test that deleted both pinned and un-pinned and non-existent blocks and made sure that the valid, un-pinned blocks where removed.

@Kubuxu I fixed a bug in the arccache.DeleteBlock() method in bf7c5b3. It seamed a small enough issue that it wasn't worth a separate pull request.

whyrusleeping · 2016-08-17T00:08:16Z

@kevina i'm not so sure that was a bug in the arccache. I guess it depends on your perspective, if i call delete on something that doesnt exist, should it ever be a problem that it doesnt exist?

kevina · 2016-08-17T00:49:32Z

@whyrusleeping the cache changes the behavior of the DeleteBlock() method which does return an error if a key was not found, therefore I consider it a bug. The cache itself relies in DeleteBlock() returning an error if a block was not found. Also I would prefer that "block rm" returns an error if a non-existent block is deleted in case I mistyped the key (or more likely I copy and pasted a key but missed a couple of characters.) If this becomes a blocker I can remove the fix and not test for deleting a non-existent block in my test case.

@Kubuxu, your view on this?

…ocks Add test that removes a combination of pinned, valid, and non-existent blocks in one command. License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

jbenet · 2016-08-17T01:38:12Z

Clarity on what is meant is important:

"make it so X is not on my local repo" should be idempotent
"check X exists in my local repo, then remove X from my local repo" not
idempotent

In general, idempotency is a very important property to have. An op like
this should be by default idempotent. You could use a flag to ask it to
signal when a block was not found.

ipfs block rm --error-if-not-present

On Tue, Aug 16, 2016 at 20:49 Kevin Atkinson notifications@github.com
wrote:

@whyrusleeping https://github.com/whyrusleeping the cache changes the
behavior of the DeleteBlock() method which does return an error if a key
was not found, therefore I consider it a bug. The cache itself relies in
DeleteBlock() returning an error if a block was not found. Also I would
prefer that "block rm" returns an error if a non-existent block is deleted
in case I mistyped the key (or more likely I copy and pasted a key but
missed a couple of characters.) If this becomes a blocker I can remove the
fix and not test for deleting a non-existent block in my test case.

@Kubuxu https://github.com/Kubuxu, your view on this?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#2962 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAIcofL46Dx13HsK4gYETd9QPz_n2UdHks5qglqcgaJpZM4JI7L-
.

kevina · 2016-08-17T01:49:45Z

@jbenet the behavior of ipfs block rm is consistent with that of the the rm command. If a block was not found it will complain but still remove any (non-pinned) blocks that are found.

In other words block rm is idempotent, the only difference will be the final status code. All blocks that it can remove will always be removed.

whyrusleeping · 2016-08-17T02:00:18Z

Oh, my bad, I read the diff backwards.

On Tue, Aug 16, 2016, 18:49 Kevin Atkinson notifications@github.com wrote:

@jbenet https://github.com/jbenet the behavior of ipfs block rm is
consistent with that of the the rm command. If a block was not found it
will complain but still remove any (non-pinned) blocks that are found.

In other words block rm is idempotent, the only difference will be the
final status code. All blocks that it can remove will always be removed.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
#2962 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABL4HCF2qmJls6Xyv_X59yL0DDxmSZ-Tks5qgmi7gaJpZM4JI7L-
.

whyrusleeping · 2016-08-17T02:16:38Z

@jbenet agreed that idempotency is important, But we need to clearly define the expectations of the datastore interface. Currently, all the datastore implementations will return ErrNotFound when delete is called on a non-existent key. If we want to change that, we need to go through and ensure that every datastore implementation behaves the same way.

whyrusleeping · 2016-08-17T02:18:00Z

This LGTM. I'll wait for @Kubuxu to review as well

jbenet · 2016-08-17T03:12:49Z

The datastore is not the ipfs blocks command. The datastore Delete wants
to give you more information.

I guess you could do what rm does and add -f
On Tue, Aug 16, 2016 at 22:18 Jeromy Johnson notifications@github.com
wrote:

This LGTM. I'll wait for @Kubuxu https://github.com/Kubuxu to review as
well

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
#2962 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAIcoQBp9r-MRXWAfNHX-BAWPO4sNpByks5qgm9ZgaJpZM4JI7L-
.

jbenet · 2016-08-17T03:13:05Z

rm -f
On Tue, Aug 16, 2016 at 23:12 Juan Benet juan2@benet.ai wrote:

The datastore is not the ipfs blocks command. The datastore Delete wants
to give you more information.

I guess you could do what rm does and add -f
On Tue, Aug 16, 2016 at 22:18 Jeromy Johnson notifications@github.com
wrote:

This LGTM. I'll wait for @Kubuxu https://github.com/Kubuxu to review
as well

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
#2962 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAIcoQBp9r-MRXWAfNHX-BAWPO4sNpByks5qgm9ZgaJpZM4JI7L-
.

whyrusleeping · 2016-08-17T03:49:06Z

@jbenet okay, i thought we were having a discussion about the changes made to the Delete method.

License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

kevina · 2016-08-17T05:37:04Z

@jbenet I added a "-f" option.

I am of the strong opinion that by default "block rm" should report some sort of error when a block is not found. Doing a quick Google "idempotent delete" brought up this stack overflow question: http://stackoverflow.com/questions/4088350/is-rest-delete-really-idempotent. Multiple people state that idempotency is about side effects not the status code, that is "you can send the request more than once without additional changes to the state of the server".

whyrusleeping · 2016-08-18T20:27:24Z

Alright, this all LGTM. Thanks a bunch @kevina !

daviddias · 2016-08-19T08:53:11Z

kevina added the status/in-progress In progress label Jul 10, 2016

kevina mentioned this pull request Jul 10, 2016

RFC: Rework Pinning interface #2963

Closed

whyrusleeping reviewed Jul 13, 2016
View reviewed changes

kevina force-pushed the kevina/block-rm branch from 36a6f93 to 7b85b57 Compare July 13, 2016 20:01

kevina mentioned this pull request Aug 9, 2016

Handling "broken" pins. #3069

Closed

kevina force-pushed the kevina/block-rm branch from 339a638 to d7940c8 Compare August 10, 2016 22:49

kevina force-pushed the kevina/block-rm branch from d7940c8 to 603fc65 Compare August 15, 2016 03:55

kevina mentioned this pull request Aug 15, 2016

ipfs block rm #2914

Closed

kevina changed the title ~~[WIP] Add "ipfs block rm" command.~~ Add "ipfs block rm" command. Aug 15, 2016

kevina added need/review Needs a review and removed status/in-progress In progress labels Aug 15, 2016

kevina force-pushed the kevina/block-rm branch 2 times, most recently from 1f158c6 to b39aff6 Compare August 15, 2016 05:49

kevina added 4 commits August 15, 2016 13:17

Add "ipfs block rm" command.

453be22

License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

"block rm": use channel instead of pipe / don't abort on non-fatal error

6ad497b

License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

"block rm": remove the option to ignore pins

c88b4b0

License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

kevina force-pushed the kevina/block-rm branch from b39aff6 to c88b4b0 Compare August 15, 2016 17:18

Fix bug in arccache.DeleteBlock() method.

bf7c5b3

License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

kevina force-pushed the kevina/block-rm branch from c278b2c to 9ae24c3 Compare August 16, 2016 23:51

"block rm": test case for removing pinned, valid, and non-existent bl…

92f6747

…ocks Add test that removes a combination of pinned, valid, and non-existent blocks in one command. License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

kevina force-pushed the kevina/block-rm branch from 9ae24c3 to 92f6747 Compare August 17, 2016 00:55

"block rm": add "--force" and "--quiet" option

8679af7

License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

kevina mentioned this pull request Aug 18, 2016

More intelligent blockstore garbage collection #3092

Open

whyrusleeping merged commit 10048ce into master Aug 18, 2016

whyrusleeping deleted the kevina/block-rm branch August 18, 2016 20:27

RichardLitt mentioned this pull request Aug 18, 2016

Open go-ipfs changes ipfs-inactive/http-api-spec#117

Closed

4 tasks

ghost mentioned this pull request Dec 23, 2016

Code Review v0.4.4...v0.4.5 #3534

Closed

kevina mentioned this pull request Feb 20, 2017

block rm: change "force" option to "ignore" #3709

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add "ipfs block rm" command. #2962

Add "ipfs block rm" command. #2962

kevina commented Jul 10, 2016 •

edited

Loading

RichardLitt commented Jul 11, 2016

whyrusleeping Jul 13, 2016

kevina Jul 13, 2016 •

edited

Loading

whyrusleeping commented Aug 4, 2016

kevina commented Aug 10, 2016

whyrusleeping commented Aug 10, 2016

kevina commented Aug 15, 2016

daviddias commented Aug 16, 2016

whyrusleeping commented Aug 16, 2016

kevina commented Aug 16, 2016

whyrusleeping commented Aug 17, 2016

kevina commented Aug 17, 2016

jbenet commented Aug 17, 2016

kevina commented Aug 17, 2016

whyrusleeping commented Aug 17, 2016

whyrusleeping commented Aug 17, 2016

whyrusleeping commented Aug 17, 2016

jbenet commented Aug 17, 2016

jbenet commented Aug 17, 2016

whyrusleeping commented Aug 17, 2016

kevina commented Aug 17, 2016

whyrusleeping commented Aug 18, 2016

daviddias commented Aug 19, 2016

Add "ipfs block rm" command. #2962

Add "ipfs block rm" command. #2962

Conversation

kevina commented Jul 10, 2016 • edited Loading

RichardLitt commented Jul 11, 2016

whyrusleeping Jul 13, 2016

Choose a reason for hiding this comment

kevina Jul 13, 2016 • edited Loading

Choose a reason for hiding this comment

whyrusleeping commented Aug 4, 2016

kevina commented Aug 10, 2016

whyrusleeping commented Aug 10, 2016

kevina commented Aug 15, 2016

daviddias commented Aug 16, 2016

whyrusleeping commented Aug 16, 2016

kevina commented Aug 16, 2016

whyrusleeping commented Aug 17, 2016

kevina commented Aug 17, 2016

jbenet commented Aug 17, 2016

kevina commented Aug 17, 2016

whyrusleeping commented Aug 17, 2016

whyrusleeping commented Aug 17, 2016

whyrusleeping commented Aug 17, 2016

jbenet commented Aug 17, 2016

jbenet commented Aug 17, 2016

whyrusleeping commented Aug 17, 2016

kevina commented Aug 17, 2016

whyrusleeping commented Aug 18, 2016

daviddias commented Aug 19, 2016

kevina commented Jul 10, 2016 •

edited

Loading

kevina Jul 13, 2016 •

edited

Loading