Fix/32/pr ports from go-ipfs to go-mfs #49

nitishm · 2018-12-25T16:32:44Z

#32 is partially completed by this PR.

Personally this doesn't seem like the best way to do things, by forcing updateChildEntry even on stateFlushed but currently that is the only way to unblock waitPub which is blocking in the FlushPath call.

A better solution will be based on the completion of #47 which should eliminate waitPub

Fixes #50.
Fixes ipfs/kubo#4514.

schomatis · 2018-12-25T16:35:59Z

Agreed, let's wait on #47.

nitishm · 2018-12-25T16:49:52Z

A better idea would be a new state (stateCreated).
Even better idea would be to store the state somewhere other than fd to make it work with the repub (maybe in the File struct ?). Will have to check where else can we store the state.

schomatis · 2018-12-25T19:31:51Z

A better idea would be a new state (stateCreated).

SGTM

Even better idea would be to store the state somewhere other than fd to make it work with the repub (maybe in the File struct ?). Will have to check where else can we store the state.

It makes sense, but we only allow one RW file descriptor (and potentially many RO descriptors), so in a way we are centralizing the file state there.

nitishm · 2018-12-25T20:02:48Z

One of the test cases randomly fails with a 0 being passed to Int63n.
Any ideas why this happens @schomatis ?

panic: invalid argument to Int63n
goroutine 37 [running]:
math/rand.(*Rand).Int63n(0xc00009a210, 0x0, 0xbe73c0)
	/home/travis/.gimme/versions/go/src/math/rand/rand.go:111 +0x17d
math/rand.Int63n(...)
	/home/travis/.gimme/versions/go/src/math/rand/rand.go:319
github.com/ipfs/go-mfs.actorWriteFile(0xc0001c4c00, 0x5, 0x3)
	/home/travis/gopath/src/github.com/ipfs/go-mfs/mfs_test.go:643 +0x1af
github.com/ipfs/go-mfs.testActor(0xc00003e6d0, 0x32, 0xc0000329c0)
	/home/travis/gopath/src/github.com/ipfs/go-mfs/mfs_test.go:698 +0x201
created by github.com/ipfs/go-mfs.TestMfsStress

See Jenkins results https://travis-ci.org/ipfs/go-mfs/builds/472151928?utm_source=github_status&utm_medium=notification

schomatis · 2018-12-25T21:28:35Z

The function seems to panic if it gets an n <= 0 (https://golang.org/pkg/math/rand/#Int63n), are we correctly reporting the created file size?

go-mfs/mfs_test.go

Line 633 in 64e7cdb

s, err := fi.Size()

Also, TestMfsStress is a heavily concurrent function, so any errors there are normally linked to a possible data race.

nitishm · 2018-12-25T22:10:54Z

Yeah that's where I am stuck. Need to figure out why my changes would affect the file creation since almost all of them are concerned only with RW flags for the fd's. Bummer.

Will try to dig in a little further. Gotta spend some time looking at the test helper functions.

fd.go

nitishm

The function seems to panic if it gets an n <= 0 (https://golang.org/pkg/math/rand/#Int63n), are we correctly reporting the created file size?

go-mfs/mfs_test.go

Line 633 in 64e7cdb
s, err := fi.Size()

Also, TestMfsStress is a heavily concurrent function, so any errors there are normally linked to a possible data race.

What does this method do ? I am completely lost here.

https://github.com/nitishm/go-mfs/blob/3191540a82040b2d124ab561fcfd971e590819b6/mfs_test.go#L506-L533

fd.go

nitishm · 2018-12-26T00:53:14Z

The failure in TestMfsStress seems to be surfacing, at random, due to this issue #50.

@schomatis If the proposal makes sense I will create a PR for it and get it into mfs_test.go ASAP.

schomatis · 2018-12-26T01:13:27Z

The failure in TestMfsStress seems to be surfacing, at random

That entails that this should also happen in master, which I haven't seen before, so before merging anything please create first a PoC PR when TestMfsStress fails without the logic added in this PR, that is, feel free to increase the number of goroutines and iterations in the test to values that can trigger this panic.

fd.go

schomatis · 2018-12-26T20:58:22Z

I'm not sure this PR is sitting on top of all the commits of #45, @nitishm could you check that?

schomatis · 2018-12-27T00:44:25Z

#45 is merged now.

nitishm · 2018-12-27T00:55:01Z

#45 is merged now.

In that case do I need to do anything ?

schomatis · 2018-12-27T01:01:46Z

In that case do I need to do anything ?

No, just rebase to (the new) master if needed.

nitishm · 2018-12-27T02:19:34Z

@schomatis Do you think we should get this in even without #47 being merged in since it bypasses the waitPub issue in any case ?

nitishm · 2018-12-27T02:23:44Z

The only commit remaining from #32 is ipfs/kubo@f4dc9a4

schomatis · 2018-12-27T14:24:58Z

@schomatis Do you think we should get this in even without #47 being merged in since it bypasses the waitPub issue in any case ?

Yes, if it makes sense on its own, we can unblock it.

The only commit remaining from #32 is ipfs/go-ipfs@f4dc9a4

We should definitely add this since it was testing the new modifications (at that time) and it was failing.

nitishm · 2018-12-27T18:27:15Z

We should definitely add this since it was testing the new modifications (at that time) and it was failing.

I merged those changes and all the tests pass, but it takes close to 500 seconds to run the 5000 iterations in TestConcurrentWriteAndFlush (with the new changes to writeFile, with the Truncate, Seek, etc ops added).

You think I should bring the iterations down to a 100 (0.063s) or 500 (0.881s) or 1000 (2.710s) vs 5000 (491.51s) maybe ?

I will add the commit once I know what to set the value to.

schomatis · 2018-12-27T20:10:22Z

ouch, that's a lot, yes, let's keep it under 1-2 seconds, it's a bit suspicious that a single writeFile iteration takes 100 ms, are we waiting for the republisher or something like that?

Also, your scale doesn't seem very proportionate, 5000 iterations costs 500 seconds, but dropping it to 1000 lowers it to 2.7? and 500 (1/10th of the original) is run in less than a second?

All that being said, if the tests are passing we're already on (a very good) track, we'll just have to fine tune some things. I'll give the PR a proper review in the following days.

schomatis · 2018-12-30T20:45:33Z

@nitishm Not very important, but could you try rebasing this PR onto the current master when you get a chance, I think that will help with the review, commit 1bbc52d has already been merged but it's still showing up in this PR (I guess because there are other commits after that not included here), getting everything in sync should clear the diff.

Pulled changes from ipfs/kubo#4517, on top of, ipfs#45. Change added to unblock the `waitPub()` call. With the elimination of stateSync cause a `updateChildEntry` to happen for `stateFlushed` as well, causing it to propogate upwards to the parent(s) [fullSync] and force a publish to happen, hence unblocking `waitPub`.

Added a new state for a freshly opened `fileDescriptor`. In `flushUp` only bubble up update if state is either `stateDirty` or `stateCreated`. `stateFlushed` should prevent bubble up.

Add 1 to rand.Intn() to prevent 0 size reader for being created.

During `flushUp()` of an un-flushed fd, be it with freshly created or modified content, always update the file descriptors inode, with the contents of the node.

Pull in commit from go-ipfs commit ipfs/kubo@f4dc9a4 Dropped nloop to 500 (from 1000) in `TestConcurrentWriteAndFlush` to reduce testing time. All unittests pass.

nitishm · 2018-12-30T23:31:34Z

@nitishm Not very important, but could you try rebasing this PR onto the current master when you get a chance, I think that will help with the review, commit 1bbc52d has already been merged but it's still showing up in this PR (I guess because there are other commits after that not included here), getting everything in sync should clear the diff.

@schomatis I did something. I don't know if I did the right thing, to do the rebase.

Does this sound right ? From nitishm/go-mfs repo I did -

1. git remote add go-mfs https://github.com/ipfs/go-mfs.git
2. git fetch go-mfs
3. git rebase go-mfs/master
4. git pull
5. git push

Does that sound right ? I dont have too much experience with dealing with forks and keeping them up to date.

schomatis · 2018-12-31T01:54:57Z

git rebase --onto master 2642dbf^ 40c7e34 -i   # <-- Check that those are actually the commits you want!
git checkout -B fix/32/pr-ports
git log master.. # Check the log before overwriting the PR.
git push # -f (to overwrite.)

You'd normally want to use the --onto option when rebasing (otherwise git does some weird stuff like rebasing things on top of themselves), I usually use the interactive -i option to check that git is about to do what I expect it to.

nitishm · 2018-12-31T02:16:55Z

So do I need to undo any of this or does it look right to you ?

nitishm · 2018-12-31T03:15:20Z

git rebase --onto master 2642dbf^ 40c7e34 -i   # <-- Check that those are actually the commits you want!
git checkout -B fix/32/pr-ports
git log master.. # Check the log before overwriting the PR.
git push # -f (to overwrite.)
You'd normally want to use the --onto option when rebasing (otherwise git does some weird stuff like rebasing things on top of themselves), I usually use the interactive -i option to check that git is about to do what I expect it to.

I managed to undo my last merge (hurray), and to rebased to the upstream/master. But it seems like my branch is already up to date with all the commits (atleast I can find 1bbc52d in my git log).

schomatis

Niiiiice. I'll leave the PR open for a couple of days in case @Stebalien wants to take a look at it (since we've been applying his code), particularly TestConcurrentWrites, since that patch came from a WIP PR that may have been unfinished.

nitishm · 2019-01-02T19:58:23Z

SGTM ! @schomatis Got anything else you want me to look at, apart from the go-ipfs docs ?

schomatis · 2019-01-02T20:16:45Z

Nothing important, we should prioritize the docs, they are also a great way for you to know what's out there in terms of the code base and to find other components you may be interested in.

Stebalien

LGTM.

Stebalien · 2019-01-08T05:44:57Z

since that patch came from a WIP PR that may have been unfinished.

The WIP part was that I couldn't fix ipfs/kubo#4519.

schomatis · 2019-01-08T12:39:07Z

Great work @nitishm!! 🎉

nitishm · 2019-01-08T14:59:02Z

Thanks for all the help @schomatis . I thoroughly appreciate it !

Stebalien · 2019-01-08T17:59:04Z

Yes, thank you both!

nitishm requested a review from schomatis as a code owner December 25, 2018 16:32

schomatis added the status/blocked Unable to be worked further until needs are met label Dec 25, 2018

schomatis assigned nitishm Dec 25, 2018

schomatis reviewed Dec 25, 2018

View reviewed changes

fd.go Outdated Show resolved Hide resolved

nitishm commented Dec 26, 2018

View reviewed changes

fd.go Outdated Show resolved Hide resolved

nitishm mentioned this pull request Dec 26, 2018

r := io.LimitReader(rread, int64(77*rand.Intn(123))) can create a 0 sized reader causing unittest(s) to panic and fail #50

Closed

schomatis reviewed Dec 26, 2018

View reviewed changes

fd.go Show resolved Hide resolved

schomatis removed the status/blocked Unable to be worked further until needs are met label Dec 27, 2018

nmalhotra added 4 commits December 30, 2018 18:25

Add stateCreated as default fd state

1cf41ee

Added a new state for a freshly opened `fileDescriptor`. In `flushUp` only bubble up update if state is either `stateDirty` or `stateCreated`. `stateFlushed` should prevent bubble up.

Add 1 to rand func in actorMakeFile test helper

0baeab2

Add 1 to rand.Intn() to prevent 0 size reader for being created.

Update unflushed fd's inode during flushUp

1c80713

During `flushUp()` of an un-flushed fd, be it with freshly created or modified content, always update the file descriptors inode, with the contents of the node.

nmalhotra added 2 commits December 30, 2018 18:25

Moved all RW ops between same locks in flushUp

ed06dbe

Merge unit test changes from issue ipfs#32

40c7e34

Pull in commit from go-ipfs commit ipfs/kubo@f4dc9a4 Dropped nloop to 500 (from 1000) in `TestConcurrentWriteAndFlush` to reduce testing time. All unittests pass.

nitishm force-pushed the fix/32/pr-ports branch from 76befbb to 40c7e34 Compare December 31, 2018 02:59

schomatis mentioned this pull request Dec 31, 2018

Review go-ipfs PR 4517 and see what can be extracted here. #32

Closed

4 tasks

schomatis approved these changes Dec 31, 2018

View reviewed changes

schomatis requested a review from Stebalien January 5, 2019 23:12

Stebalien approved these changes Jan 8, 2019

View reviewed changes

Stebalien merged commit 1989254 into ipfs:master Jan 8, 2019

nitishm mentioned this pull request Jan 8, 2019

Set fileDescriptor.closed=true in Close() method. #46

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix/32/pr ports from go-ipfs to go-mfs #49

Fix/32/pr ports from go-ipfs to go-mfs #49

nitishm commented Dec 25, 2018 •

edited by schomatis

Loading

schomatis commented Dec 25, 2018

nitishm commented Dec 25, 2018 •

edited

Loading

schomatis commented Dec 25, 2018

nitishm commented Dec 25, 2018

schomatis commented Dec 25, 2018

nitishm commented Dec 25, 2018

nitishm left a comment

nitishm commented Dec 26, 2018

schomatis commented Dec 26, 2018

schomatis commented Dec 26, 2018

schomatis commented Dec 27, 2018

nitishm commented Dec 27, 2018

schomatis commented Dec 27, 2018

nitishm commented Dec 27, 2018

nitishm commented Dec 27, 2018

schomatis commented Dec 27, 2018

nitishm commented Dec 27, 2018

schomatis commented Dec 27, 2018

schomatis commented Dec 30, 2018

nitishm commented Dec 30, 2018

schomatis commented Dec 31, 2018

nitishm commented Dec 31, 2018

nitishm commented Dec 31, 2018

schomatis left a comment

nitishm commented Jan 2, 2019

schomatis commented Jan 2, 2019

Stebalien left a comment

Stebalien commented Jan 8, 2019

schomatis commented Jan 8, 2019

nitishm commented Jan 8, 2019

Stebalien commented Jan 8, 2019

Fix/32/pr ports from go-ipfs to go-mfs #49

Fix/32/pr ports from go-ipfs to go-mfs #49

Conversation

nitishm commented Dec 25, 2018 • edited by schomatis Loading

schomatis commented Dec 25, 2018

nitishm commented Dec 25, 2018 • edited Loading

schomatis commented Dec 25, 2018

nitishm commented Dec 25, 2018

schomatis commented Dec 25, 2018

nitishm commented Dec 25, 2018

nitishm left a comment

Choose a reason for hiding this comment

nitishm commented Dec 26, 2018

schomatis commented Dec 26, 2018

schomatis commented Dec 26, 2018

schomatis commented Dec 27, 2018

nitishm commented Dec 27, 2018

schomatis commented Dec 27, 2018

nitishm commented Dec 27, 2018

nitishm commented Dec 27, 2018

schomatis commented Dec 27, 2018

nitishm commented Dec 27, 2018

schomatis commented Dec 27, 2018

schomatis commented Dec 30, 2018

nitishm commented Dec 30, 2018

schomatis commented Dec 31, 2018

nitishm commented Dec 31, 2018

nitishm commented Dec 31, 2018

schomatis left a comment

Choose a reason for hiding this comment

nitishm commented Jan 2, 2019

schomatis commented Jan 2, 2019

Stebalien left a comment

Choose a reason for hiding this comment

Stebalien commented Jan 8, 2019

schomatis commented Jan 8, 2019

nitishm commented Jan 8, 2019

Stebalien commented Jan 8, 2019

nitishm commented Dec 25, 2018 •

edited by schomatis

Loading

nitishm commented Dec 25, 2018 •

edited

Loading