Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: gypkg for GYP dependency management #7240

Closed
indutny opened this issue Jun 9, 2016 · 21 comments
Closed

Proposal: gypkg for GYP dependency management #7240

indutny opened this issue Jun 9, 2016 · 21 comments
Labels
build Issues and PRs related to build files or the CI.

Comments

@indutny
Copy link
Member

indutny commented Jun 9, 2016

Hello @nodejs/build @nodejs/collaborators @nodejs/ctc !

I have been recently working on gypkg, which is something that I have craved for a very long time. What do you think about trying it out for Node.js core, and maybe libuv (cc @saghul) ?

This will allow us to put every GYP dependency to a separate repo, and manage them with a semver in a node.gyp file. Should be pretty neat, and should make the upgrade process easier and cleaner.

Thank you,
Fedor.

@indutny indutny added the build Issues and PRs related to build files or the CI. label Jun 9, 2016
@indutny
Copy link
Member Author

indutny commented Jun 9, 2016

Oh, and also it will reduce future size of the Node.js repo tree, since we won't be adding way too much changes on each dep update anymore.

@indutny indutny changed the title Use gypkg for GYP dependencies management Proposal: gypkg for GYP dependencies management Jun 9, 2016
@indutny indutny changed the title Proposal: gypkg for GYP dependencies management Proposal: gypkg for GYP dependency management Jun 9, 2016
@eljefedelrodeodeljefe
Copy link
Contributor

I like the work vey much, but as posted on twitter I am very strongly opposed to introducing another js-foreign dependency or something that increasing organizational dependency to one.

I think it's rather time to build our own tooling since we are big enough and can agree that most stuff out there isn't really what we are looking for out of the box and we are just piggy-backing on the Google stuff.

Since node is proliferated really much nowadays I think we would also do the software world a favor and build more cross platform tooling that anyone could use for other purposes too without having things like python installed correctly. Gyp seems orthogonal to that goal.

I am working on something wrapping shelling out to gcc, cl.exe and clang and to be honest that seems very easy to me and I don't even see the need for more advanced build system features. Everything else, like dependency management, change detection, target management, scripting can be done very easily if not much better with JS as opposed to declarative configs and declarative Makefile scripts.

@saghul
Copy link
Member

saghul commented Jun 9, 2016

Ohai! Here are my 2 cents, FWIW.

This project is nice in that it reduces bolireplate and makes it easier to use GYP for Node and other projects. Alas, it still requires Google's GYP, which doesn't change much of the status quo.

If we backtrack a little, it seems that the dependency on Python was one of the things people don't seem to like. Not to mention the fact that GYP doesn't work on Python 3, which adds more to the confusion. Google itself it transitioning to GN, which just generates Ninja buildfiles, and lets Ninja do the work. Personally, I think this a good approach.

My little contribution to help solve this problem was GYN a while ago. It's basically Google's GYP, with a single generator (Ninja) and ported to Python 3. I could compile Node itself. It was met with some resistance since it's still Python, which I do understand.

That said, I think a good way forward would be to port GYP to JS but make it just generate only Ninja files. We'd still be using GYP files, which Node is kinda heavily invested on it by now, and Ninja is super small and self-contained, so bundling it in shouldn't be much of a problem.

I briefly mentioned this last part to @eljefedelrodeodeljefe on Twitter. The idea being that Ninja could be replaced later, saving that stage and going directly from GYP files to binaries if need be. Personally, I'd take the Unix approach of one tool one job and let Ninja do the compilation.

One thing I wonder though, is that if this looks like a bigger problem than it is because we don't have a good story for binary packages. If we did then only the people who build the package would need the build tools, and (specially those on Windows) others wouldn't have much trouble.

@seishun
Copy link
Contributor

seishun commented Jun 9, 2016

This will allow us to put every GYP dependency to a separate repo, and manage them with a semver in a node.gyp file.

I don't think this is a good thing. Currently, Node.js repo is self-contained. This guarantees that even 10 years from now you can easily check out an ancient commit and build it. If it had dependencies in other repos, then it would depend on whether these repos still exist.

@jbergstroem
Copy link
Member

I generally share @saghul's point of view. Adding on top: I now and then attempt to disconnect gn from the rest of the google stack which is perfectly possible but very hard in reality without hard forking; it's just moving too quick. It's relatively simple to build a standalone version of it; and I really like the idea of using gn-files/ninja as the "future" native compilation stack but I have a hard time seeing it happen if we stick with python.

@rvagg
Copy link
Member

rvagg commented Jun 9, 2016

Some of this discussion can move to nodejs/CTC#2 which is attempting to take a broader perspective on what to do with GYP in core and the ecosystem.

@kzc
Copy link

kzc commented Jun 9, 2016

@saghul

I think a good way forward would be to port GYP to JS but make it just generate only Ninja files.

It's a chicken and egg problem. I think people should be able to build node without a node binary. So not using JS to build is ironically preferable IMO.

@saghul
Copy link
Member

saghul commented Jun 9, 2016

@kzc We could use Duktape to bootstrap the process. It's a single C file, so maybe that would be acceptable? (just as a fallback) Or write the replacement in C++.

@kzc
Copy link

kzc commented Jun 9, 2016

@saghul

We could use Duktape to bootstrap the process

Very interesting idea. The issue is that Duktape doesn't support most node standard modules like fs AFAIK, although it does have some Buffer compatibility. Could be confusing maintaining JS code that is not compatible with node.

@saghul
Copy link
Member

saghul commented Jun 9, 2016

Yes, it doesn't support any modules out of the box. It does have a Buffer implementation, though it has the 0.12 API. We really only need to read and write a file, so adding a couple of builtins should do. At any rate, I don't think the bootstraping is a big deal, comparing it with the rest.

@indutny
Copy link
Member Author

indutny commented Jun 9, 2016

FWIW, I'm working on JS port of GYP right now. Will keep you updated.

@eljefedelrodeodeljefe
Copy link
Contributor

@kzc chicken-and-egg problem does not apply in this case. It's a common misconception. That is what cross compilation and downloading binaries is for. Go can build itself, so can gcc. We are no different.

@kzc
Copy link

kzc commented Jun 9, 2016

@eljefedelrodeodeljefe Comparisons with Go don't apply as node can't be built with node alone. A C++ compiler is really the key prerequisite of building node. That's what builds v8, libuv, openssl and node itself. That's what I like about @saghul's proposal bootstrapping via Duktape - it eliminates the python dependency in favor of the C++ dependency that node can't avoid.

@eljefedelrodeodeljefe
Copy link
Contributor

@kzc that's wrong. Any build tool needs a C/C++ compiler. In the end, no matter what tool you are using, you are calling out to your platforms compiler binary. So why using a dependency for making a call to gcc when you can do it with child_process?

@eljefedelrodeodeljefe
Copy link
Contributor

@kzc fun fact: the reason why you want to use some toolchain is to avoid writing something like the below just to compile an addon.cc. But you can do it. Same goes for running make -j8 and copy pasting and output line when you compile node in order to mirror gyps (and any other build tool's behaviour)

  "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\amd64\CL.exe" /c /I"C:\Program Files (x86)\Windows Kits\10\Include\10.0.10240.0\ucrt" /I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\atlmfc\include" /I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include" /I"C:\Users\jefe-win\.node-gyp\6.2.1\include\node" /I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include" /I"C:\Users\jefe-win\.node-gyp\6.2.1\src" /I"C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140" /I"C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0" /I"C:\Program Files (x86)\Windows Kits\8.1\Include\um" /I"C:\Program Files (x86)\Windows Kits\8.1\Include\shared" /I"C:\Program Files (x86)\Windows Kits\8.1\Include\winrt" /I"C:\Users\jefe-win\.node-gyp\6.2.1\deps\uv\include" /I"C:\Users\jefe-win\.node-gyp\6.2.1\deps\v8\include" /I..\node_modules\nan /Zi /nologo /W3 /WX- /Ox /Ob2 /Oi /Ot /Oy /GL /D NODE_GYP_MODULE_NAME=addon /D WIN32 /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _HAS_EXCEPTIONS=0 /D BUILDING_V8_SHARED=1 /D BUILDING_UV_SHARED=1 /D BUILDING_NODE_EXTENSION /D _WINDLL /GF /Gm- /MT /GS /Gy /fp:precise /Zc:wchar_t /Zc:forScope /Zc:inline /GR- /Fo"Release\obj\addon\\" /Fd"Release\obj\addon\vc140.pdb" /Gd /TP /wd4351 /wd4355 /wd4800 /wd4251 /errorReport:queue /MP ..\addon.cc

@kzc
Copy link

kzc commented Jun 9, 2016

Any build tool needs a C/C++ compiler.

Great. Then we're in agreement on that point.

Using node to bootstrap node is a needless complication. It can be done, but it's not worth the trouble when other ways are simpler.

@eljefedelrodeodeljefe
Copy link
Contributor

-.- I bail the conversation here.

@jbergstroem
Copy link
Member

@kzc the relevant question here isn't bootstrapping node itself, rather being able to compile node native modules on a system where node is installed.

@kzc
Copy link

kzc commented Jun 9, 2016

the relevant question here isn't bootstrapping node itself, rather being able to compile node native modules on a system where node is installed.

@jbergstroem Ah, okay. My mistake.

But perhaps this JS port of GYP initiative could have applications to bootstrapping node itself if successful.

@indutny
Copy link
Member Author

indutny commented Jun 12, 2016

Port of GYP (WIP): https://github.com/indutny/gyp.js

@Trott
Copy link
Member

Trott commented Jul 7, 2017

This issue has been inactive for sufficiently long that it seems like perhaps it should be closed. Feel free to re-open (or leave a comment requesting that it be re-opened) if you disagree. I'm just tidying up and not acting on a super-strong opinion or anything like that.

@Trott Trott closed this as completed Jul 7, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build Issues and PRs related to build files or the CI.
Projects
None yet
Development

No branches or pull requests

8 participants