ipfs pin rm _very_ slow & cpu intensive

[XertroV]

Version information:

go-ipfs version: 0.4.13-
Repo version: 6
System version: amd64/darwin
Golang version: go1.9.2

Type:

Bug

Description:

I had about 57k pinned items (all small files) totaling about 30MB.

Due to #4673 I figured I'd try and remove them since I don't actually need them pinned all the time (lots of testing atm).

Currently running a script to get all pinned MHs and call ipfs.pin.rm via js-ipfs-api on them. It's running at 5-10 MH/s and IPFS is using 120-150% CPU (modern macbook pro 13", retrieved CPU % via top)

The node process is using about 1% CPU.

I'd be nice to have a ipfs pin rm --all feature as well.

Took a profile while the pin rming was going on too: QmcVtZCuvMk1dynoF9pZ5cpNi5ausm4XWRG7MGkZTm94ZQ

[XertroV]

Update: after removing all pinned MHs IPFS is using about 5% cpu while adding 2-3 MH/s, much more reasonable.

[Stebalien]

Yeah, it's the same bug. Any modification to a large pin set will take ages.

[Voker57]

Did you hit unpinned CIDs on your way by any chance? This currently causes VERY expensive computation.

[XertroV]

@Voker57 - do you mean were my pinned CIDs linked to unpinned CIDs?
In that case: no. The pinned CIDs were all standalone objects with no links to other CIDs.

When I rm'd the pinned CIDs I used this script: https://gist.github.com/XertroV/1c909efeb061193a8feb4f4a2531153b - it gets the fulled list of pinned CIDs then runs through and calls ipfs pin rm individually on each.

[Voker57]

@XertroV no I meant were you calling pin rm on on objects that are not pinned. Looks like that shouldn't have happened in your case.

[Stebalien]

Closing in favor of #5221.