'node-gyp rebuild' installation error when adding turkserver to a meteor app

[ZJoveski]

Hi Andrew,

I was wondering if you could help me with the following problem. Today I was trying to add turkserver to my Meteor application, but I kept getting this error:

C:\Users\PC4all\Meteor\exp_cs>meteor add mizzao:turkserver

mizzao:turkserver: updating npm dependencies -- request, libxmljs,
validator, querystring, async, deepmerge...
gypnpm ERR! Windows_NT 6.1.7601
npm ERR! argv
"C:\Users\PC4all\AppData\Local.meteor\packages\meteor-tool\1.1.3\mt-os
.windows.x86_32\dev_bundle\bin\node.exe"
"C:\Users\PC4all\AppData\Local.meteor\packages\meteor-tool\1.1.3\mt-os
.windows.x86_32\dev_bundle\bin\node_modules\npm\bin\npm-cli.js"
"install" "libxmljs@0.8.1"
npm ERR! node v0.10.36
npm ERR! npm v2.7.3
npm ERR! code ELIFECYCLE

npm ERR! libxmljs@0.8.1 install: node-gyp rebuild
npm ERR! Exit status 7
npm ERR!
npm ERR! Failed at the libxmljs@0.8.1 install script 'node-gyp rebuild'.
npm ERR! This is most likely a problem with the libxmljs package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node-gyp rebuild
npm ERR! You can get their info via:
npm ERR! npm owner ls libxmljs
npm ERR! There is likely additional logging output above.

=> Babel active on file extensions: es6.js, es6, jsx
=> Errors while adding packages:

While building package mizzao:turkserver:
error: couldn't install npm package

I am developing my Meteor application on a Windows 7 machine and from what I have been able to gather from posts on forums, the error seems to be related to that fact (I did not find any posts describing the same problem on Linux machines).

Any help would be greatly appreciated!

Thanks,
Zlatko

[mizzao]

This also seems to be referenced here: atom/atom#2754, and is surely an interaction between npm and Windows. I'm not sure if Meteor is using an official Windows build of npm but it might help for you to check.

Also, it seems like you're designing a graph coloring experiment. I have done that as well :-)

[ZJoveski]

Thank you for your prompt response! I tried the suggestions on the link that you provided, but they did not seem to work. I have also tried configuring the python path and MS Visual Studio version for npm as suggested here: https://www.robertkehoe.com/2015/03/fix-node-gyp-rebuild-error-on-windows/, but that did not help either. My guess is that those commands only affect the npm version that comes with node.js installed on my computer, but not the npm used by Meteor.

I am not quite sure how to check if Meteor is using an official Windows build and how that would help.

And yes, we are designing a graph coloring coordination experiment and we are hoping to use turkserver for managing our MTurk experiments.

[mizzao]

@ZJoveski I would file a bug in the Meteor repo about this and provide reproduction examples: https://github.com/meteor/meteor/issues

I think it's great that you are using d3 and Meteor to build a graph coloring experiment. Four years ago I implemented a graph coloring experiment with very old technologies that I would never use today :-)

What university are you from?

[ZJoveski]

Thanks for the suggestion! I will try that too!

I'm from Vanderbilt University.

[mizzao]

So we just ran into this exact error on a Linux (i.e. not Windows) machine and it went away when we installed npm system-wide rather than using the Meteor bundled version.

My suspicion is that Meteor's bundled npm might be out of date or something, and that it defers to a system wide npm when it exists. So install that and give it a spin again. You might want also to reference this when you report the bug.

Who's your advisor at Vanderbilt?

[mizzao]

After a little more digging, pretty sure this is due to npm not having the Python path configured correctly: meteor/meteor#4802 (comment)

[ZJoveski]

I already had npm installed on my computer (as part of node.js). I tried reinstalling node.js, but that didn't help. Also, I have Python 2.7.6 installed. The PYTHONPATH environment variable is set to C:\Python27\python.exe, and I had already tried running

npm config set python C:\Python27\python.exe

but again, I keep getting the same error.

[ZJoveski]

I will try to do a better job of referencing issues/discussions opened on different sites in the future. Sorry about that. And I hope you don't mind me posting the same question on stackoverflow. But currently, nothing of what I've tried seems to work and I was just hoping that someone could identify this as a more general problem with Meteor on Windows.

[ZJoveski]

Oh, and, I am working on a project with Prof. Yevgeniy Vorobeychik.

[mizzao]

You could try forking TurkServer and using a new version of libxmljs, although it's possible it might break the mturk library - which is why I haven't tried going and updating all those yet as it could be a whole new can of worms.

Alternatively, install a Linux VM using something like VirtualBox on your Windows machine, and you can happily use this software just like I am :)

[mizzao]

@ZJoveski any luck with this? I am thinking of updating libxmljs to remove this binary dependency entirely.

Also, check out http://ldworkin.github.io/turkserver-tutorial/ which might help you get started!

[ZJoveski]

No, I just could not figure it out. So, as you suggested, I switched to a Ubuntu VM and I have not had any of the same problems.

I have been going through Lili's code at https://github.com/ldworkin/meteor-prisoners-dilemma for the past week and a half (I found it a bit easier to follow than the CrowdMapper example). I noticed the tutorial over the weekend, but by then I think I have figured out most of the things included there.

Right now I am working on a customized assigner for the recruitment session because we want people to go through a description of the game, a consent form, and in addition play a few practice games.

I had one question (I am not sure if I should open a separate issue for it). Since we plan on deploying our application on https://modulus.io and not on https://www.meteor.com, is there a simple way to avoid passing the settings.json file as a --settings argument when deploying the app, but somehow have the app find and read the contents of the file on its own? Unlike meteor deploy, modulus deploy does not allow passing of arguments from the command line.

[ldworkin]

I'll chime in because I ran into this same problem -- the (annoying) way to do this on Modulus is to add a METEOR_SETTINGS environment variable and set it equal to the entire settings JSON blob, eg: https://www.dropbox.com/s/r93gi92e6ws1lii/meteor_settings.PNG?dl=0

I think there's also a way to hardcode settings into the app, but I can't remember at the moment.

Let me know if you have any questions about my messy and undocumented code!

[mizzao]

I've never used Modulus but I know they have a few tools for easing Meteor deployments. So modulus deploy uses demeteorizer, right?

I agree CrowdMapper is not a great example. That's why @ldworkin has been so helpful in providing the tutorial and a hello world app! I am also thinking about how to overhaul the assigner API; it is a bit too low-level right now.

Let me know if it would be helpful to discuss how to set up the logistics of your experiment (apart from TurkServer). It can be a bit confusing for first timers, as there are many pitfalls to avoid.

[ZJoveski]

Thanks @ldworkin! That actually doesn't seem too annoying. And thank you for both your code and the tutorial!

@mizzao, it would be great if you could spare some time to discuss the logistics of our experiment! I could certainly use some advice.

[ZJoveski]

Oh and, yes, I think modulus deploy uses demeteorizer.

[mizzao]

Hey @ZJoveski, do you want to set up a call with me and Yevgeniy sometime this week or next week?

[ZJoveski]

Sure, that would be great! I will send you an email at mizzao@gmail.com.

[mizzao]

So FYI, and for future reference: I've seen this error both on Linux and Windows machines and it seems the culprit is a missing Python path: nodejs/node-v0.x-archive#4047

Solution in the short term is to either make sure Python is accessible by node-gyp or use a Linux VM or OS X where this isn't an issue. The long term solution I think would be to migrate off the somewhat outdated https://github.com/jefftimesten/mturk and use something newer like https://github.com/JoseBarrios/mturk-api, although I haven't had the chance to test out the latter yet.

[mizzao]

The master branch now doesn't use the https://github.com/jefftimesten/mturk API with the submodule, which basically gets rid of all the cross-platform issues with node-gyp. I've tested all the basic MTurk functions with the tutorial app and everything seems to work fine.

If you're still having this issue, you may want to try the latest commit on master - let me know if there are any problems.