Graceful stop #205

richardschneider · 2018-02-17T03:46:53Z

Use api/v0/shutdown to kill a daemon process
Get tests working on windows

See ipfs/js-ipfs#1192 for background details.

Note that this is blocking on ipfs/js-ipfs#1224. When fixed, change the package.json to use the latest ipfs.

richardschneider · 2018-02-18T18:31:38Z

daviddias · 2018-02-18T19:23:03Z

package.json

@@ -100,7 +100,7 @@
    "detect-port": "^1.2.2",
    "dirty-chai": "^2.0.1",
    "go-ipfs-dep": "0.4.13",
-    "ipfs": "^0.27.5",
+    "ipfs": "github:ipfs/js-ipfs",


Let's hold this PR until we have a new release of js-ipfs (it can happen soon).

It's here, can't wait to get this in :)

codecov · 2018-02-21T23:46:43Z

Codecov Report

❗ No coverage uploaded for pull request base (master@4cc69a0). Click here to learn what that means.
The diff coverage is 90%.

@@            Coverage Diff            @@
##             master     #205   +/-   ##
=========================================
  Coverage          ?   86.92%           
=========================================
  Files             ?       17           
  Lines             ?      673           
  Branches          ?        0           
=========================================
  Hits              ?      585           
  Misses            ?       88           
  Partials          ?        0

Impacted Files	Coverage Δ
src/factory-in-proc.js	`86.95% <100%> (ø)`
src/ipfsd-daemon.js	`91.15% <77.77%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4cc69a0...36aa183. Read the comment docs.

richardschneider · 2018-02-22T00:17:08Z

@dryajov I've rebased on you changes. The config hack really speeds things up!

Still waiting on an js-ipfs release that has a shutdown route. Maybe @diasdavid will give us some love.

Could you look into these issues

The spawn tests are failing when testing on the browser
Still get port 9999 issue on macos

dryajov · 2018-02-22T18:43:08Z

@richardschneider I'm trying to figure out why the tests are failing now, but having been able to do so yet. As for the port issue, we were able to determine that it's not related to ipfsd-ctl itself, but rather something running on those ports on mac os - #209.

richardschneider · 2018-02-22T18:49:55Z

@dryajov cheers!

dryajov · 2018-02-22T22:50:31Z

The issue seems to be somehow related to the version entry missing from IndexDB on first run. I can't debug it, since running the set of tests in the browser a second time around works fine, it only seems to fail on first run. Here is the error:

Skipping the version test allows the rest of the tests to pass.

dryajov · 2018-02-26T18:35:21Z

@diasdavid @victorbjelkholm any idea of what might be causing this - #205 (comment)?

richardschneider · 2018-03-01T20:37:31Z

@diasdavid Thanks for the ipfs v0.28 release. This PR is now ready for review and merge.

daviddias · 2018-03-01T21:45:37Z

@richardschneider all CI is failing. Errors:

Let's be attentive of everyone's time and request a review or a merge only when the code is fully ready.

JonKrone · 2018-03-08T00:20:56Z

This happens on proc nodes during the ipfsd.version() test because the IPFS instance it creates to get the version hasn't finished booting. There's a fix for this in my PR that addresses some ipfsd initialization bugs. @dryajov This is the same error you posted earlier.

This issue is basically the same thing at a different point - proc nodes aren't waiting for their IPFS instance to finish booting before attempting to work with them. Here's the fix.

Those initialization changes should address the CI issues here and then I think we've got green-green-green IPFSD CI! 😁

JonKrone · 2018-03-08T16:44:18Z

@richardschneider @dryajov

Jenkins and Travis are green ✔️

richardschneider · 2018-03-08T21:39:11Z

@diasdavid Is this now ready for a merge?

daviddias · 2018-03-09T00:15:52Z

src/factory-in-proc.js

+    const ipfs = new IPFS(options)
+    ipfs.once('ready', () => {
+      ipfs.version((err, _version) => {
+        if (err) { callback(err) }


needs a return. Otherwise chaos will happen.

Agree, extremely bad code!!!

daviddias · 2018-03-09T00:16:09Z

src/ipfsd-daemon.js

   *
   * @param {function()} callback - Called when the process was killed.
   * @returns {undefined}
   */
  killProcess (callback) {
    // need a local var for the closure, as we clear the var.
+    const self = this


Self is not needed.

Disagree, it is needed in subprocess.once('exit', ...

this should not be used in a lambdaexpression! See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/Arrow_functions

daviddias

Still not finished. CI is also not always green.

daviddias · 2018-03-11T23:04:06Z

src/factory-in-proc.js

+      ipfs.version((err, _version) => {
+        if (err) { return callback(err) }
+        callback(null, _version)
+      })


Is this starting the node? I'm sure this is causing repo lock just to check the version.

Hey @dryajov, curious about this as there is a TODO to fix this on the go/js factory. Did the mentioned solution hit roadblocks originally? Can we make an issue and work on it separate from this PR?

daviddias · 2018-03-11T23:04:47Z

src/factory-in-proc.js

-        ? node.start(options.args, cb)
-        : cb()
-    ], (err) => callback(err, node))
+    node.exec.once('ready', () => {


Something isn't quite right here. the node is created but then what we listen on the ready event is exec??

dryajov · 2018-03-12T21:53:22Z

I'm reworking proc nodes to propagate the ready event. That is the only reliable way of ensuring that the node is fully constructed when interacting with it. Ideally, the user doesn't have to care about it as it would be taken care of by the factory.

richardschneider · 2018-03-12T22:30:38Z

@dryajov Seems that this PR is being re-purposed. Can we get some closure?

dryajov · 2018-03-12T23:16:20Z

This change is required to get all CIs green. I'm fine with making the change in a different PR but this would depend on it. I'll create a separate PR.

richardschneider · 2018-03-12T23:27:21Z

@dryajov If it's need then please add it here. I'm just concerned that changes are creeping in here that have nothing to do with stopping.

dryajov · 2018-03-13T06:20:16Z

jenkins is failing because of:

daviddias · 2018-03-13T08:45:29Z

@dryajov Travis still fails with a bunch of timeouts

daviddias · 2018-03-13T20:48:24Z

src/factory-in-proc.js

+      node.init((err) => {
+        if (err) { return callback(err) }
+        node.version(callback)
+      })


It is very odd to me that we start a new instance of IPFS to get the version. This will create a memory leak + it will lock the repo for a consequent node spawn. @dryajov can you expand why to do it this way?

@diasdavid I'm not aware of any other way to get the version - is there a better one you can suggest?

btw, the instance is not started, it is simply initialized with a repo, to start it we would have to call node.start.

@dryajov are you saying that a repo is created to just get the version of ipfs? Sound like the node.version should work offline/un'init for the info it can provide, namely, the impl version.

If it was absolutely impossible (which it isn't), at least there should be a cleanup step here.

Also, I thought about just using ipfs-repo directly to read the version, but it turns out we still need to initialize a repo to get the version, since there is no guarantee that the default repo in ~/.jsipfs corresponds to the version of the IPFS in our exec param.

Well, it turns out that the version is read by the repo here - https://github.com/ipfs/js-ipfs/blob/494da7f8c5b35f491f22a986ff5e8c456cc0e602/src/core/components/version.js#L14...L24. I think that's an overkill and we should separate the repo version from the ipfs version, that way we don't need a repo. If noone objects I can go ahead and make that change in IPFS.

There is a need for a Repo version and IPFS version, that is correct. It should be ok to get the IPFS version without the Repo existing though.

Agreed - I'm making the change in IPFS to allow for that.

daviddias · 2018-03-13T20:51:53Z

src/ipfsd-in-proc.js

@@ -68,6 +72,8 @@ class Node {
      EXPERIMENTAL: this.opts.EXPERIMENTAL,
      libp2p: this.opts.libp2p
    })
+
+    this.exec.once('ready', () => this.emit('ready'))


Why use a different pattern than the one used for ipfs-daemon (new Node + node.start)?

I believe not doing that can end up in race conditions - take a look at maybeOpenRepo, its called by the IPFS constructor through boot(self), but it is async, and the only way of knowing when it finished is by hooking into the ready event.

What I'm weird'ed out is that in ipfsd-in-proc, the IPFS instance starts (and therefore does network activity, lock repo, etc) on the constructor, while in ipfsd-daemon, the daemon only starts when .start is called. This asymmetry will lead to confusion and debugging issues in the future.

ah, I see - we're not actually starting the node since IPFS is constructed with both, init and start as false, take a look here - https://github.com/ipfs/js-ipfsd-ctl/blob/a61fa50d6834e61465398cd1d54e60ec913358eb/src/ipfsd-in-proc.js#L67...L74. This was done specifically to keep that symmetry with the other types of daemons, the ready event is there to make sure we don't trip over anything.

I see. Makes sense then. Thanks for clarifying :)

dryajov · 2018-03-13T23:08:47Z

So the issue ended up being browser only. By default, if the repo doesn't exist, the static repo version is used, but in the browser the error returned was not being matched correctly, hence to version was being returned at all. The fix in ipfs/js-ipfs#1262, should address that.

daviddias · 2018-03-14T03:27:24Z

Thank you @dryajov :) I'll release a patch version of js-ipfs as soon as I have decent Internet again (npm is not friendly of in-flight wifi)

daviddias · 2018-03-14T14:20:29Z

Seems that it is all good \o/

Mac OS fails because Jenkins doesn't remember how to install npm anymore //cc @victorbjelkholm

ghost assigned richardschneider Feb 17, 2018

ghost added the status/in-progress In progress label Feb 17, 2018

richardschneider requested a review from daviddias February 17, 2018 03:47

This was referenced Feb 17, 2018

feat: add keysize through an option to spawn #203

Merged

Updating CI files ipfs/js-ipfs#1103

Merged

daviddias reviewed Feb 18, 2018

View reviewed changes

richardschneider mentioned this pull request Feb 23, 2018

Make spawning super fast! #190

Closed

daviddias mentioned this pull request Feb 23, 2018

fix: try with a different port #211

Merged

dryajov mentioned this pull request Feb 23, 2018

feat: rework http-api tests to use ipfsd-ctl ipfs/js-ipfs#1234

Merged

victorb mentioned this pull request Feb 23, 2018

repo.lock does not delete automatic after closing ipfs-desktop ipfs/ipfs-desktop#610

Closed

dryajov mentioned this pull request Mar 6, 2018

fix: daemon should recognize previously created repos. #212

Merged

ghost assigned dryajov Mar 8, 2018

richardschneider added 8 commits March 7, 2018 18:44

feat: use api/v0/shutdown to stop a deamon

00958a7

feat: use api/v0/shutdown to stop a deamon

ec58831

test: work on windows

1db2ae4

test: timeout issues

6ecc02a

fix: timeout

252e1d5

test: run on windows

c3315ac

chore: force a build

67a2041

chore: update dependency on ipfs

8cd5abe

daviddias requested changes Mar 9, 2018

View reviewed changes

dryajov added 2 commits March 8, 2018 20:49

fix: return callback on error

162ffb8

fix: use this instead of self

36fbb7a

daviddias requested changes Mar 11, 2018

View reviewed changes

dryajov added 3 commits March 12, 2018 23:49

fix: correctly initialize proc nodes

5fc58bb

fix: no need for setImmediate

b1f5538

fix: simplify version call

a61fa50

JonKrone mentioned this pull request Mar 13, 2018

Feat/pin tracking issue ipfs/js-ipfs#1261

Closed

6 tasks

fix: timeouts

2d3cf40

daviddias requested changes Mar 13, 2018

View reviewed changes

fix: no need to init repo to get version

3ffcee3

fix: remove singleRun

4ad247b

chore: update deps

36aa183

ghost assigned daviddias Mar 14, 2018

daviddias approved these changes Mar 14, 2018

View reviewed changes

daviddias merged commit 359dd62 into master Mar 14, 2018

ghost removed the status/in-progress In progress label Mar 14, 2018

daviddias deleted the graceful-stop branch March 14, 2018 14:20

vmx mentioned this pull request Apr 17, 2018

Enable Bitswap tests on Windows ipfs/js-ipfs#1313

Closed

Graceful stop #205

Graceful stop #205

Conversation

richardschneider commented Feb 17, 2018

richardschneider commented Feb 18, 2018

daviddias Feb 18, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Feb 21, 2018 • edited Loading

Codecov Report

richardschneider commented Feb 22, 2018

dryajov commented Feb 22, 2018

richardschneider commented Feb 22, 2018

dryajov commented Feb 22, 2018

dryajov commented Feb 26, 2018

richardschneider commented Mar 1, 2018

daviddias commented Mar 1, 2018

JonKrone commented Mar 8, 2018 • edited Loading

JonKrone commented Mar 8, 2018

richardschneider commented Mar 8, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daviddias left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dryajov commented Mar 12, 2018

richardschneider commented Mar 12, 2018

dryajov commented Mar 12, 2018

richardschneider commented Mar 12, 2018

dryajov commented Mar 13, 2018

daviddias commented Mar 13, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dryajov Mar 13, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dryajov commented Mar 13, 2018

daviddias commented Mar 14, 2018

daviddias commented Mar 14, 2018

daviddias Feb 18, 2018 •

edited

Loading

codecov bot commented Feb 21, 2018 •

edited

Loading

JonKrone commented Mar 8, 2018 •

edited

Loading

dryajov Mar 13, 2018 •

edited

Loading