Rewrite pinning to store pins in IPFS objects #1225

tv42 · 2015-05-11T20:21:36Z

Still needs a migration. Notes for that:

There's two parts:

1009aeb
The keys for that looked like /local/pins/{direct,recursive,indirect}/keys/* and can be just removed.
1673b9c
The keys for that look like /local/pins/{direct,recursive,indirect}/keys (parents of the above), contain json, and can be fed to the new pinning mechanism to generate the new state.
The direct and recursive json are arrays, the indirect json is a dict {key: refcount}.

tv42 · 2015-05-13T02:03:10Z

Travis failures look like OOM and timeouts.

@chriscool When you say "all", that makes me think "out of scope for this issue" ;)

@whyrusleeping @jbenet Let me know if there's anything unclear here, or something I can help with. I'm basically waiting to hear back from you two.

whyrusleeping · 2015-05-13T04:12:39Z

pin/pin.go

 	PinWithMode(util.Key, PinMode)
+	// RemovePinWithMode is for manually editing the pin structure.
+	// Use with care! If used improperly, garbage collection may not
+	// be successful.
 	RemovePinWithMode(util.Key, PinMode)


Yeah, youre probably right about moving these together.

i'd think the other way was safer for users, but shrug

whyrusleeping · 2015-05-13T04:51:56Z

A few things that are confusing to me, but other than that, this looks really nice. Removes a bunch of things i wanted removed, and makes bunch of change towards a more sensible pinning setup

chriscool · 2015-05-13T05:48:50Z

@tv42 what I wanted to say is that as far as I know the other file comparison functions, like test_cmp, test_sort_cmp, are supposed to take the expected file first and the actual file second, and it would be nice if the comparison functions you introduce were following the same pattern too.

tv42 · 2015-05-13T14:18:19Z

@chriscool Hmm. Sharness's test_cmp is documented to take arguments in the order expected, actual. The preexisting code was test_sort_cmp gc_out_actual2 gc_out_exp2, and I just kept that order. You're right, this is probably worth fixing.

chriscool · 2015-05-13T14:42:14Z

@tv42 great! Thanks for fixing it!

jbenet · 2015-05-15T02:19:52Z

pin/pin.go

+	}
+	rootKey := util.Key(rootKeyI.([]byte))
+
+	ctx := context.TODO()


does this need to be wired up?

Yes, but that changes the Pinner API -> later.

jbenet · 2015-05-15T03:48:14Z

@tv42 minor comments above o/ -- aside, this LGTM.

vitzli · 2015-05-17T03:48:37Z

Debian Stretch inside virtualbox, 2 GB RAM, 2 CPU cores allocated for VM (Core 2 Quad 9550).

HW config 1

VM is located on HDD. Guest root partition and guest /srv partition are located on separate physical HDDs (both SATA2). Test files located at /srv/test1, IPFS_PATH points to /srv/ipfs_conf.

user@deb-stretch-ipfs:/srv/test1$ time sha256sum random.dat
2dc9095156bf916fc1cd07d98bf9e0d330cfbefedf9f037a65e378cbc9e70ab1  random.dat

real    0m19.488s
user    0m13.504s
sys 0m5.732s

sha256sum speed: 102.6 MB/s

Adding urandom junk to ipfs:

user@deb-stretch-ipfs:/srv/test1$ time ipfs add random.dat 
added QmfQDSXnxvjVPDzhwmHK83CWFaCUwyYLTnjEf5dZm1wJqb random.dat

real    3m39.620s
user    1m10.676s
sys 0m43.760s

Adding file full of 0xFF:

user@deb-stretch-ipfs:/srv/test1$ time ipfs add notsorandom.dat 
added QmNmKeTsXeeLHA8TYeZg6DyWx96hhfSiqWBsoSM7wec3ck notsorandom.dat                                     

real    0m26.844s
user    0m22.704s
sys 0m11.288s

Average speed: 74.6 MB/s, iotop log

vitzli · 2015-05-17T05:09:13Z

HW config 2 (host OS)

Core 2 Q9550, 4 CPU cores, 8 GB RAM, test file and ipfs_path on the same SSD.
Adding random.dat:

added QmfQDSXnxvjVPDzhwmHK83CWFaCUwyYLTnjEf5dZm1wJqb random.dat

real    5m11.659s
user    0m53.792s
sys 0m8.120s

Progress bar finishes at 10:55:15, process ends around 10:58, iotop log, average speed: 6.43 MB/s

HW config 3 (host OS)

IPFS_PATH on (fragmented?) HDD, test file on SSD:
Doing ipfs add gave estimated time around 10 minutes for the same 2GB file, I assume that it is caused by the fragmentation of HDD partition (20 GB left out of 300GB). Also, there were very slow read/write speeds in iotop:

11:04:17  1895 be/4 alex        7.98 K/s 3112.68 K/s  0.00 %  1.50 % ipfs add random.dat
11:04:17  1895 be/4 alex        7.98 K/s 3113.39 K/s  0.00 %  1.49 % ipfs add random.dat
11:04:18  1895 be/4 alex        0.00 K/s 3631.49 K/s  0.00 %  1.70 % ipfs add random.dat

I would discard this test.

whyrusleeping · 2015-05-17T05:17:41Z

@vitzli nice! so this appears to be a little bit faster than master, right?

vitzli · 2015-05-17T16:42:58Z

@whyrusleeping I'm sure now that I've made a mistake, I commented on it in #1216, results are somewhat disappointing, pin branch seems to be slower than master, but I need somebody to verify my results.
1 core, random.dat file, pin branch, test file and ipfs dir on the same separate HDD:

TEST 1  3m13.276s
TEST 2  3m12.529s
TEST 3  3m24.626s
TEST 4  3m31.363s
TEST 5  3m44.372s
TEST 6  3m49.319s
TEST 7  3m43.692s
TEST 8  4m3.791s
TEST 9  3m42.013s
TEST 10 3m41.873s

Average time: 3m36.6s, 9.2 MB/s
2 cores:

TEST 1  3m39.991s
TEST 2  3m47.141s
TEST 3  3m50.335s
TEST 4  3m56.028s
TEST 5  4m2.825s
TEST 6  4m5.213s
TEST 7  3m57.563s
TEST 8  4m5.529s
TEST 9  4m0.147s
TEST 10 4m2.940s

Average time 3m56.7s, 8.5 MB/s

whyrusleeping · 2015-05-17T19:29:24Z

@vitzli hrm.. thats what @tv42 said might be the case. a little disappointing... but I'm sure there is plenty of room for optimization.

Could you maybe put together a script to run these benchmarks? would be great to have a 'ipfs-add-benchmark' thing around.

tv42 · 2015-05-17T19:55:06Z

Round 1 of the pinning rework was explicitly done so it doesn't change the APIs much. Any other way would have been too much chaos in the codebase.

whyrusleeping · 2015-05-17T20:42:03Z

So further revisions wont require any sort of a migration, correct?

tv42 · 2015-05-17T21:21:34Z

It's always possible we discover the format was bad. I'd be surprised if that happened soon.

tv42 · 2015-05-18T20:38:31Z

Uhh, I added more tests and may or may not have found a bug. Please don't merge while I diagnose. Preparing migration code should still be safe.

Pinner had method GetManual that returned a ManualPinner, so every Pinner had to implement ManualPinner anyway.

Platform-dependent behavior is not nice, and negative refcounts are not very useful.

WARNING: No migration performed! That needs to come in a separate commit, perhaps amended into this one. This is the minimal rewrite, only changing the storage from JSON(+extra keys) in Datastore to IPFS objects. All of the pinning state is still loaded in memory, and written from scratch on Flush. To do more would require API changes, e.g. adding error returns. Set/Multiset is not cleanly separated into a library, yet, as it's API is expected to change radically.

tv42 · 2015-06-09T03:52:23Z

I saw it too, now. Looking..

tv42 · 2015-06-09T04:14:46Z

@jbenet @whyrusleeping That's a dagmodifier error branch that explicitly unpins an object.

        // We found the correct child to write into
        if cur+bs > offset {
            // Unpin block
            ckey := key.Key(node.Links[i].Hash)
            dm.mp.RemovePinWithMode(ckey, pin.Indirect)

            child, err := node.Links[i].GetNode(dm.ctx, dm.dagserv)
            if err != nil {
// THIS TRIGGERS
                return "", false, err
            }
            k, sdone, err := dm.modifyDag(child, offset-cur, data)
            if err != nil {
                return "", false, err
            }

//NEVER GETS HERE
            // pin the new node
            dm.mp.PinWithMode(k, pin.Indirect)

tv42 · 2015-06-09T04:16:30Z

Well, the error at that point is a missing block too -- but that's the cause of the other missing block, that's what the test really craps out on. Lovely.

tv42 · 2015-06-09T04:27:40Z

Found it.

…lure

tv42 · 2015-06-09T04:47:56Z

Should be good again. @whyrusleeping FYI the dagmodifier changes.

whyrusleeping · 2015-06-09T14:49:04Z

@tv42 good catch, thanks!

whyrusleeping · 2015-06-16T23:50:10Z

merged into dev0.4.0

tv42 · 2015-07-15T15:26:57Z

Can this be closed now?

jbenet · 2015-07-16T00:23:56Z

I think so yep :) 👍 thanks @tv42

jbenet added the status/in-progress In progress label May 11, 2015

whyrusleeping reviewed May 13, 2015
View reviewed changes

jbenet reviewed May 15, 2015
View reviewed changes

whyrusleeping mentioned this pull request May 16, 2015

Slow ipfs add #1216

Closed

tv42 and others added 11 commits June 8, 2015 20:49

blocks/set: Remove now-unused NewDBWrapperSet

deceff0

Simplify Pinner interface by folding ManualPinner into Pinner

0a3399e

Pinner had method GetManual that returned a ManualPinner, so every Pinner had to implement ManualPinner anyway.

pin: Remove dead code

2b75aff

pin: Remove double bookkeeping of refcount keys

892352c

Use uint64 for indirect pin refcounts

70edb24

Platform-dependent behavior is not nice, and negative refcounts are not very useful.

Typo

ae89354

sharness: Don't assume we know all things that can create garbage

08ab857

sharness: Don't assume we know all things that can create garbage

9bb1b55

pin: Future-proof against refcount marshaled size changes

8ab2b79

bump fsrepo version to 3

47473d6

tv42 added 3 commits June 8, 2015 21:45

pin: Do not accidentally delete indirect pins on Flush

9b85b9c

dagmodifier: Don't lose pin if old and new key happen to be equal

070f128

dagmodifier test: Add TODO note about how bad luck can cause test fai…

2a453f2

…lure

tv42 force-pushed the pin branch from 397fbcf to 2a453f2 Compare June 9, 2015 04:46

wking mentioned this pull request Jun 18, 2015

need separation of private blocks and public blocks #1386

Open

jbenet closed this Jul 16, 2015

jbenet removed the codereview label Jul 16, 2015

kevina mentioned this pull request Aug 18, 2016

More intelligent blockstore garbage collection #3092

Open

whyrusleeping deleted the pin branch August 18, 2016 05:10

aschmahmann mentioned this pull request Dec 1, 2021

Release v0.11 #8343

Closed

80 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rewrite pinning to store pins in IPFS objects #1225

Rewrite pinning to store pins in IPFS objects #1225

tv42 commented May 11, 2015

tv42 commented May 13, 2015

whyrusleeping May 13, 2015

jbenet May 15, 2015

whyrusleeping commented May 13, 2015

chriscool commented May 13, 2015

tv42 commented May 13, 2015

chriscool commented May 13, 2015

jbenet May 15, 2015

tv42 May 15, 2015

jbenet commented May 15, 2015

vitzli commented May 17, 2015

vitzli commented May 17, 2015

whyrusleeping commented May 17, 2015

vitzli commented May 17, 2015

whyrusleeping commented May 17, 2015

tv42 commented May 17, 2015

whyrusleeping commented May 17, 2015

tv42 commented May 17, 2015

tv42 commented May 18, 2015

tv42 commented Jun 9, 2015

tv42 commented Jun 9, 2015

tv42 commented Jun 9, 2015

tv42 commented Jun 9, 2015

tv42 commented Jun 9, 2015

whyrusleeping commented Jun 9, 2015

whyrusleeping commented Jun 16, 2015

tv42 commented Jul 15, 2015

jbenet commented Jul 16, 2015

Rewrite pinning to store pins in IPFS objects #1225

Rewrite pinning to store pins in IPFS objects #1225

Conversation

tv42 commented May 11, 2015

tv42 commented May 13, 2015

whyrusleeping May 13, 2015

Choose a reason for hiding this comment

jbenet May 15, 2015

Choose a reason for hiding this comment

whyrusleeping commented May 13, 2015

chriscool commented May 13, 2015

tv42 commented May 13, 2015

chriscool commented May 13, 2015

jbenet May 15, 2015

Choose a reason for hiding this comment

tv42 May 15, 2015

Choose a reason for hiding this comment

jbenet commented May 15, 2015

vitzli commented May 17, 2015

HW config 1

vitzli commented May 17, 2015

HW config 2 (host OS)

HW config 3 (host OS)

whyrusleeping commented May 17, 2015

vitzli commented May 17, 2015

whyrusleeping commented May 17, 2015

tv42 commented May 17, 2015

whyrusleeping commented May 17, 2015

tv42 commented May 17, 2015

tv42 commented May 18, 2015

tv42 commented Jun 9, 2015

tv42 commented Jun 9, 2015

tv42 commented Jun 9, 2015

tv42 commented Jun 9, 2015

tv42 commented Jun 9, 2015

whyrusleeping commented Jun 9, 2015

whyrusleeping commented Jun 16, 2015

tv42 commented Jul 15, 2015

jbenet commented Jul 16, 2015