What about gyp?

[jbergstroem]

Edit: For all people landing here because you have issues after updating from nodejs to io.js or getting compile errors while building native modules: Your issue is about v8 being updated, not gyp itself. Please file issues at the module's issue trackers about having issues with newer v8 versions instead

Some months ago, I brought up the fact that gyp is pretty much abandonware on the nodejs mailing-list. People chipped in with concerns and questions, but it pretty much stayed dormant after that. I think the outcome of "ignoring" it – especially since the io.js (and/or node) community would now have to cater to gyp development if needed – is not what I'd call planning for a stable future.

I feel that the road to 1.0.0 would be best spent on trying to come to a conclusion as to what role gyp should play in the future of io.js. I'm inclined to summarise what's been said in the previous topic, but will refrain from doing so since it could come off as biased (go read it, now). If gyp will stay around, a decision to keep it instead of "for now" would be optimal. The idea of python requiring ruby to build, the same way io.js requires python is somewhat silly to me though.

[rvagg]

I think the logical path is to move to gn in sync with Chromium, it's something we could start experimenting with already here.

[jbergstroem]

@rvagg: The problem is that the gn devs has shown little to no interest in making gn accessible outside the chromium sources. I've yet to revisit this, but at least last time I was looking at it there was a very tight coupling with headers and includes left and right. Replacing gyp with gn would also be my first suggestion, but if it requires some forking action we might not be much better off than sticking with gyp (besides sans Python) from a maintainer point of view.

[indutny]

I'm a bit afraid of GN, honestly saying. I think we should either stay with GYP, or move to something completely new and public. Probably that thing would compile to ninja, so it'll be a GN alternative in our control :)

[bnoordhuis]

I agree with @indutny: we should either fork gyp (and gut everything from it except make/ninja/VS support) or switch to another build system. It pains me to say it but perhaps the gyp critics were right in 2011 and we should have switched to cmake instead.

@jbergstroem Thanks for filing that issue. Now we know what the position of the GN developers is. We could perhaps lobby hard for a standalone build but there is no guarantee of success and the burden of maintenance would fall upon us. Maintaining a gyp fork where we don't have to care about other users is much easier.

[bnoordhuis]

A tangentially related issue: I think StrongLoop would be willing to donate developer time to replace gyp and node-gyp with a native application that doesn't require a python install. Windows users without python and RHEL 5 users with ancient python versions are the bane of our support department.

[jbergstroem]

@bnoordhuis That's great news. I think that having python as a buildtime requirement for io.js but not within the npm ecosystem would be an interesting part-goal. As for the bigger picture, getting something to work seamlessly over multiple os:es is a major headache, so starting fresh should barely even be considered.

I guess one option I haven't seen mentioned yet (for npm/gyp modules) could be falling back to basic Makefiles with something similar to how pg_config work for the postgres community. It's tried and tested.

I think the guys behind npm should weigh in here as well if possible - their input would be very valuable. gyp is somewhat stuck between io/node.js and npm and both sides could benefit from a well-defined surface (as well as who is responsible for what).

[bnoordhuis]

I guess one option I haven't seen mentioned yet (for npm/gyp modules) could be falling back to basic Makefiles with something similar to how pg_config work for the postgres community. It's tried and tested.

I'm unsure how that would work on Windows. Bundling make.exe with npm is probably not a big deal but writing a Makefile that works with cl.exe on Windows and g++/clang++ on UNIX is likely not a pleasant experience. We could predefine recipes but then we're back to creating our own ad hoc build system.

I think the guys behind npm should weigh in here as well if possible - their input would be very valuable. gyp is somewhat stuck between io/node.js and npm and both sides could benefit from a well-defined surface (as well as who is responsible for what).

That would be @isaacs and @othiym23 and perhaps @mmalecki too now.

Maybe @TooTallNate as well, he maintains node-gyp.

[maxkorp]

Rather than looking at suddenly having to maintain a standalone GN for the global community, what about just forking it, the same way everyone is talking about forking GYP? That way it's still an io.js specific build system, but there's already got a much faster system than gyp as a foundation.

[indutny]

I have an idea, could we write this tool in subset of javascript that is capable of compiling to some C code? Or in C that could be easily transpiled to the JS?

This would simplify life of node-gyp significantly. Otherwise after @bnoordhuis proposal, it'll need to download platform and arch -specific binaries on every install, or build them.

[bnoordhuis]

@maxkorp Two reasons that I can think of:

1. GN stands for 'Generate Ninja'. It doesn't support Visual Studio or Xcode or even make.
2. GN is closely integrated with the massive chromium-base source tree. It's very hard to try and extract (and maintain) a slimmed down version from that.

I tried and didn't exactly fail but it was no picnic either (3.5 MB source-only is the smallest I could trim it down to, in case you're wondering.) I would dread every upgrade and those are bound to happen frequently because of its young age. Forking and maintaining gyp is almost certainly much less work.

@indutny

it'll need to download platform and arch -specific binaries on every install, or build them.

Yes, but I don't see a problem with that. We can include it as part of a release.

Another approach is to remake gyp in pure JS but that would preclude the possibility of building node with that tool because of the chicken and egg problem. A native binary would still need to bootstrap itself first, of course, but that's less insurmountable.

For the record, I'm not married to the idea of a native binary but it would certainly make life easier for some users.

[maxkorp]

@bnoordhuis Regarding CN, a cursory look around doesn't seem to indicate anything contrary to your experience, and I'm inclined to say you're right in wanting to avoid it.

A question to consider then, Is ninja notably slower on windows/mac than msbuild/xcode? If so, that grants a gyp fork a significant edge in my mind. If not, I'd say the the simplicity of a single type of build file is definitely something to consider regardless of the system generating them, be it something new and open as indutny suggests, or a fork of gyp, or any of the other existing options.

Regarding the pureJS option, what about bundling some minimalist interpreter like flathead.js (windows already has jsc.exe) and defaulting to that when io isn't available? You then have something capable enough for an initial build, but something more optimized for subsequent builds.

Regardless, I'll echo the sentiment that dropping the dependency on python (without introducing any others) would be a massive plus.

[jbergstroem]

@bnoordhuis I'm not well versed in Windows, but wouldn't visual studio's nmake.exe be the equivalent of make, not cl.exe?

Built on that, either io.js or npm would grow functionality similar to pg_config (pass linker, compile and version info) and possibly some other magic. Nmake should be able to tackle the Makefiles.

Having python part of iojs bootstrap/compile would be acceptable to me. Platforms like Windows would most likely not build io.js anyway - rather use a binary. Even if gyp went away, there's still ./configureand stuff in tools/ that needs to be taken care of to remove Python completely

@maxkorp Ninja and gyp are two different things, often used together (gyp generating build information that ninja executes). GN is now replacing gyp in the chromium ecosystem.

[bnoordhuis]

@maxkorp

A question to consider then, Is ninja notably slower on windows/mac than msbuild/xcode? If so, that grants a gyp fork a significant edge in my mind.

I don't know if it's faster or slower but GN doesn't know how to generate VS project files and that would likely upset Windows developers. I think the best GN can do is drive cl.exe or msbuild.exe through ninja.

Regarding the pureJS option, what about bundling some minimalist interpreter like flathead.js (windows already has jsc.exe) and defaulting to that when io isn't available? You then have something capable enough for an initial build, but something more optimized for subsequent builds.

It's an option. The interpreter would still have to be bootstrapped first but that's doable for a simple project like that. I'll have to think about it.

@jbergstroem

wouldn't visual studio's nmake.exe be the equivalent of make, not cl.exe?

Sorry, I meant driving cl.exe from make.exe. nmake is not a drop-in replacement for GNU make; forcing the use of nmake would make supporting both Windows and UNIX more complicated for add-on developers.

[rlidwka]

Bundling make.exe with npm is probably not a big deal but writing a Makefile that works with cl.exe on Windows and g++/clang++ on UNIX is likely not a pleasant experience.

Is it possible to write a substitute for make + /bin/sh + builtins in javascript? Just for windows users?

[seishun]

I'm more curious about V8's stance on this. It seems unlikely to me that they would drop gyp in favor of a Chromium-dependent build system. I would rather expect them to either take over gyp or switch to something completely different. I think we shouldn't jump the gun here and instead just wait and see what V8 does.

[jbergstroem]

@seishun v8 started moving in May. Chromium is set to replace gyp throughout their ecosystem by March, 2015 (full roadmap here). This doesn't mean that all .gyp-files needs to go away, since GN already reads them - which means that v8 might possibly stay "backwards"-compatible with gyp for an unforeseeable future.

Edit: Linked to quote about March, 2015.

[jbergstroem]

@bnoordhuis Although its not a drop-in replacement, how much stuff from gnu make does a native module really need? Do you have any worst case examples? When moving between gmake and make on bsd's, most of the "gnu make'isms" I find is usually trivial to avoid. I should probably spend some time reading up on nmake.exe before assuming anything else..

One thing that could improve cross-os compatibility could be accompanying the toolset with a linter for Makefiles which tells you about gnu make'isms.

[maxkorp]

I think the best GN can do is drive cl.exe or msbuild.exe through ninja.

I'm pretty sure you're correct here, digging around a bit. It's not clear how far along that is though, as according to the GN roadmap windows compilation hasn't started, but digging through chromiums build docs it seems like it's decently far along. The uncertainty is more than I had originally thought though, which to me reaffirms the thought that GN probably isn't the best approach, at least for the time being.

Anyways, I was thinking that ninja driving msbuild would be perfectly acceptable (you get a functional build at the end, after all), but I hadn't really considered community acceptance. Clearly VS is still a required part of the toolchain, and while VS is still obviously a huge deal in windows development, I'd be curious to see how attached Windows devs working on core/native modules are to the actual VS environment beyond just it's toolchain. While I wouldn't be too surprised to find people being ok with using over IDEs (and thereby not requiring the project files), I'd be more surprised than if we found that many would still rather develop in VS. And my experience is that if you're using VS, offloading your build to a different system is an absolute hassle (at least as of current, although it seems this might be improving a bit recently).

All of the above is moot though anyways if the end goal is trying to avoid building something from scratch.

If we were to take the leap and concoct something new, a native application doesn't seem so bad. Perhaps taking Indutny's idea of some sort of JS subset that compiles to C, and also offering it for download prebuilt? My tiny interpreter idea seems a bit risky with more thought, as then you have to find/build/port a simple interpreter to work on a new system, before you can build the desired interpreter on the new system. All you've really done is added a lizard to the chicken and the egg.

[seishun]

@jbergstroem

v8 started moving in May.

This hardly looks like moving to me. It rather looks like they're adding gn support just for Chromium.

Chromium is set to replace gyp throughout their ecosystem by March, 2015 (full roadmap here).

That page explicitly states that it only applies to Chromium and downstream of Chromium ("Upstream projects like Skia and V8 can choose to stay on GYP if they like").

[bnoordhuis]

@jbergstroem

Although its not a drop-in replacement, how much stuff from gnu make does a native module really need? Do you have any worst case examples?

I'm by no means an nmake expert but I know it doesn't support .PHONY, .SECONDEXPANSION, functions like $(shell), $(subst), etc.

It's like trying to write a Makefile that works with both GNU make and BSD make; things are sufficiently different that it's a pain and most people won't bother. You are welcome to use Makefiles - npm is relatively build system agnostic - but I wouldn't want to foist that onto every add-on developer.

[jbergstroem]

@seishun You're right; I was somewhat brisk in my assumptions. I went through their issue tracker and couldn't find anything that pointed to a planned switch (or deprecation) - so they're probably not going anywhere at the moment. I think they're using gn for their own builds though.

@bnoordhuis Sure. I just want to explore options.

[piranna]

On NodeOS we are thinking about develop a Javascript version of GNU make just to don't need to use a compiled one, and it could be also useful for Windows. Any help is welcome.

Disclaimer: I'm the lead developer of NodeOS.

[formula1]

I currently am trying to convert all of gyp into node compatible javascript. When it comes down to it, the people leading this don't have an infinite amount of money. Whether or not they are even getting funding currently is questionable. If you want to see io.js succeed, you can be apart of the solution. Though showing there are issues in the first place is being a part of the solution.

My repo is here https://github.com/formula1/node-gyp/tree/gyp-js

You can run with the following command

 in_dir=gyp/pylib out_dir=gyp-js/pylib node gyp-js/automated.js

Its certianly not perfect, nor may it ever be. However, with the files I manually did there wasn't anything crazy that node couldn't do without addons. I added a dependency on async because without it my hand written code would look like poop.

[bjouhier]

+1 for a pure JS build tool.

[bnoordhuis]

My personal preference is to clone (a subset of) gyp as a native binary, to bootstrap iojs and to build add-ons with. Ideally, it would be a single-file build: gcc -o buildtool buildtool.c

A pure JS solution has some attractive properties but only works for add-ons. We would end up with two things to maintain, gyp and gyp-js, and that seems less than ideal.

[bjouhier]

@bnoordhuis Agree. What matters is that build does not add dependencies over what we already have.

[piranna]

We can asume that anyone that want to build Node.js/Io.js has a binary of them, so the pure Javascript version would be enought, isn't it?

[jbergstroem]

FWIW, I'm with @bnoordhuis; but I'd also be ok with a pure-js version since the more important goal is the npm ecosystem (and its dependencies/maintenance), not necessarily building node itself.

[formula1]

Well that makes me feel better about feeling like the conversion is drudgery. Either way, I made another attempt with coffeescript. I also made the debugging alot better (at least for the coffee script version). But this seems like a month+ project alone if not more so I'm probably just going to leave it as is for now.

If it can be compiled to a binary form easier, sounds fine to me!

[unbornchikken]

Reading through this thread, regarding native addon build system (not to build io.js itself), there are two viable options mentioned: switching to gn or creating a pure js build system from strach (or by copying gyp functionality). Lemme share my arguments against those with ya!

• gn (or anything that based on Ninja): in a bright Linux based future, it might work, but on Windows, it makes things mutch harder than we have right now by using node-gyp. We can overcome the issue that there is no Ninja Windows distribution exists yet by putting a Windows binary in node-gn, but we loose the ability to create Visual Studio sln files from our projects. This will be a huge PITA since we have to switch from industry leading Visual Studio inegrated debugger to WinDbg console. (I'm willing to switch to my arse instead!)
• pure js build system: yeah, that's my dream too, +1000. But who should develop it? Node/io.js team? I think making an evented I/O for v8 javascript solution and making a cross platform native C/C++ build system, that has features those are comparable to gyp or better, and its first version is stable enough to drive the entire node native addon ecosystem, and is multiplatform of course is two really different things. A pioneer like @TooTallNate? Considering the previous statement, this is gonna be a complex and huge task for an invidual. A company like Joyent? I don't think that anyone here thinks that this is a viable option.

I think there is an other option: adapt an already estabilished build system for this purpose. I propose CMake. It's mature, fast, stable, has a friendly license, used natively by almost all cross platform C++ IDE-s, and of course, there is already exists a node-gyp like module to integrate it with the npm ecosystem (the author is me). It has nice features, for example it uses Ninja and Clang if present. Created module has no dependency on Python 2.x, only CMake.

I created a tutorial, to show you that creating native modules with CMake.js is as easy as creating them with node-gyp. Or easier, because since it's based on (almost standard) CMake, I can develop and debug my native modules even in stuff like Qt Creator.

I'm here because of two reasons:

• I need a high reach. Without anyone notice that there are other options available than node-gyp for creating native modules (right now), there aren't really other options available.
• I want to know the opinion of this community. If you all think that this idea is just plain stupid, then I won't put more effort into it.

[unbornchikken]

ANN: I've added nw.js support and it's Boosted.

[lygstate]

How about QBS? https://wiki.qt.io/Qt_Build_Suite
It's based on QML and like javascript.

[unbornchikken]

Yeah, I did investigate that option before decided to use CMake. For simple modules both syntaxes are quite simple (see CMake's there), like your gyp file is right now. Many of native modules are exists because those are intended to wrap an already existing C/C++ library for node. If the actual library uses other build system than your module, then at first, you have to deconstruct its project file, then build your own by hand. It's a huge PITA. So the real question is, what is the build system that is used by the most of libraries out there? That is CMake (at least compared to QMake).

But I think it's not the build system that is the achilles heel of the node platform right now. It's the requirement of installing Visual Studio / Windows SDK (12 GB hog!) on Windows just to code or host JavaScript! (There was the Python 2.x dependency but that has been addressed in CMake.js already.) I think I can address this, and I'm gathering feedback on this topic there: cmake-js/cmake-js#17

[IonicaBizau]

Is this still an issue? Using Node v4.0.0, latest version of gyp works fine for me. 🌟

[saghul]

@IonicaBizau it still doesn't work with Python 3, you must be using Python 2.

[cjntaylor]

Just to add my own thoughts to this issue, I've created my own build system for node native modules based on CMake: https://github.com/cjntaylor/node-cmake

This stems from a bunch of work I've been doing in the robotics space, where CMake is the defacto standard (especially in ROS). The key is that I needed a way to integrate node module building both into npm, and into a larger CMake build structure, without relying on the presence of a node interpreter. CMake is very powerful in its own right, and is fully capable of finding/downloading/validating the required sources for building an extension.

Beyond this, tighter coupling on CMake side makes the build more flexible for IDE integration and cross-compilation, something I do frequently. This does still provide ncmake to simplify npm integration, and invocation on the node side.

I've included an example in the repository, and I'd love any feedback and comments regarding the module. This is only meant to add to the conversation as an alternative; I share @unbornchikken's concerns about the future stability of gyp/node-gyp.

Has anything been settled on that front? Is there a better place this should be discussed?

[jbergstroem]

@cjntaylor at this point a discussion should probably live in here: https://github.com/nodejs/NG. Me and possibly some other people in the build group is interested in solving this long term as well; feel free to join and chat about it: http://gitter.im/nodejs/build

[unbornchikken]

I believe the only viable solution would be an integrated ffi and web worker thread with shared buffer support. With those, almost anything could be created in JS side by using the modern es6 runtime.

[Trott]

Any objection to closing this and directing people to the NG repo? Maybe open a stub issue there?

[jbergstroem]

Sure, lets. See above for new issue.