[FEATURE] frontend-maven-plugin - A better alternative for yeoman-maven-plugin ?

[ddewaele]

Overview of the issue

At the moment, yeoman-maven-plugin is used to handle the frontend packaging for the production profile.

Although it works it has some shortcomings :

• It depends on an existing installation of node/npm on the system
• It is impossible to choose a specific version of node/npm
• It is not really actively maintained.

Perhaps there are better alternatives, like frontend-maven-plugin that seems to have more features / a larger community / more stars on github.

In my current environment, I have very little control over my Jenkins installation.

• I cannot change the jenkins PATH and the Jenkins NodeJS plugin is kinda broken. (and doesn't play well with maven projects).
• To make matters worse, on my Jenkins machine, in /usr/bin there is an old incompatible version of NodeJS (v0.10.x) that will always be used as default by jHipster and fail the build.

This means that at the moment, I cannot release jHipster apps with the production profile, with minified JS from Jenkins.

I believe it would be better if jHipster had more control over the specific version of node/npm that is used to package the jHipster UI

Suggest a Fix

I was wondering if people were open to looking into frontend-maven-plugin as a worthy alternative for the yeoman-maven-plugin ?

I also found the CI documentation a bit confusing, as it mentioned both :

• using the Jenkins NodeJS plugin to install NodeJS.
• using a shell script to download / install NodeJS.

Having a self-contained maven plugin that doesn't depend on external dependencies (availability of node/npm or globally installed NodeJS modules on the system) will make jHipster more versatile.

A similar plugin exists for Gradle.

[jdubois]

Oh yes, at the beginning of the project we tried it with the help of @eirslett -> we had a few issues with proxies (which should be corrected now), so we can definitely have a look at migrating to this plugin again.
I agree it is more actively maintained and has more traction.

[ddewaele]

I'm testing it right now (locally and on Jenkins).
There's also a Gradle plugin that does something similar : https://github.com/srs/gradle-node-plugin

I'll post my findings here.

[ddewaele]

I'm guessing that during the prod packaging you want to do a controlled

• npm install
• bower install
• gulp build --no-notification

Is that correct ?

[ddewaele]

Just did a maven release with -Pprod and the frontend-maven-plugin on our Jenkins.

It worked fine, but the build (a simple jHipster 0.30.0 app with 4 entities) took 17minutes. Reasons being :

1. Releasing via maven includes the prepare step, so it will always double your build time.
2. npm install takes a long time (and needs to be executed twice when performing a release)

I noticed in the yeoman plugin there was no npm / bower install, just the gulp build. It assumed that gulp was already installed.

[gmarziou]

To make matters worse, on my Jenkins machine, in /usr/bin there is an old incompatible version of NodeJS (v0.10.x) that will always be used as default by jHipster and fail the build.

Have you tried using nvm?
It works well in Jenkins, just add an initial step to select the node version you want using nvm use 4.4.2.

[ddewaele]

@gmarziou

No, I didn't find nvm on the machine. (someone did try to install nvm using npm but I guess that's not the same nvm we're talking about).

• Do I need to get nvm onto the jenkins server somehow ? (using curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.31.0/install.sh | bash)
• Is there a jenkins plugin that takes care of making nvm available ?
• The initial step you are referring to becomes a pre-build shell script ?

I still like the idea of being able to build an application without having lots of external dependencies (and nvm to me is still something that needs to be installed on a machine if I understand correctly)

[gmarziou]

Do I need to get nvm onto the jenkins server somehow ? (using curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.31.0/install.sh | bash)

Yes

Is there a jenkins plugin that takes care of making nvm available ?

No

The initial step you are referring to becomes a pre-build shell script ?

Yes with a single line.

Your understanding is correct.

nvm is mostly used for setting up your dev environment but I have used it successfully with Jenkins to build an Ionic app which is a more complex task than building a JHipster angular app.

I'm OK with switching to another plugin as long as it does not increase build time.

With Nexus 3 just being released, I plan to try it to speed up my builds and dev env setup involving npm and bower.

[ddewaele]

@gmarziou It's an interesting discussion ... unfortunately you need to perform an npm install if you want to make gulp available locally.

There is a huge dependency tree in the jhipster package.json that takes > 6 minutes to download.

The current yeoman-maven-plugin also does an npm install so I'm guessing there won't be any speed impact.

mvn -Pprod clean install run on a CI server will takes considerably longer then a simple non prod mvn clean install

How long does your mvn -Pprod clean install take ?

[gmarziou]

I don't like the clean goal of JHipster pom that deletesnode_modules folder, so first thing I do is to remove this after I generate a new project. If I ever want a fully clean build in Jenkins, I just wipe ou t project's workspace but it is very rare. After all, node_modules is just about build tools in JHipster.

My Jenkins build takes about 5 minutes.

I read somewhere that Nexus's npm registry cache could reduce by 30% the duration of your npm installphase but I suppose it depends on how slow your internet connection is.

[deepu105]

IMO jhipster or the generated apps maven/gradle build should not be
responsible for maintaining the run time environment.
For example the maven build downloads its dependency once and uses the same
for a local prod build and if you run it in a CI where every build starts
on a clean VM/folder, like our travis builds, it downloads the dependency
again.

So there is no special handling for local build or CI build and
maven/gradle does its normal stuff always.

For NPM the behaviour should be the same. If there is a local node_modules
folder that should be reused without clearing it for each build. So npm
install will run faster on local build and in a clean CI/folder it will
download it anyway. This is how it works today and I guess thats a good way.
Gael FyI: I think mvn clean doesnt delete node_modules folder currently,
atleast not on windows were I checked. If it does anywhere then we should
change that IMO

So if changing the plugin alters this behaviour then personally I do not
agree. If the plugin can maintain the same behaviour then I agree.

I would happily agree with all that npm sucks :P

On Thu, 7 Apr 2016 8:09 am Gaël Marziou, notifications@github.com wrote:

I don't like the clean goal of JHipster pom that deletesnode_modules
folder, so first thing I do is to remove this after I generate a new
project.

My Jenkins build takes about 5 minutes.

I read somewhere that Nexus's npm registry cache could reduce by 30% the
duration of your npm installphase but I suppose it depends on how slow
your internet connection is.

—
You are receiving this because you are subscribed to this thread.

Reply to this email directly or view it on GitHub
#3356 (comment)

[gmarziou]

Deepu, I really think maven does clean it in prod profile, look here:

https://github.com/jhipster/generator-jhipster/blob/master/generators/server/templates/_pom.xml#L1069

[lluiscanals]

Yes, I can confirm too, I have to comment those lines in pom every clean in prod profile in order not to delete the node_modules folder, this change can be exported to 2.27.2 too.

[deepu105]

@gmarziou you are right I checked on dev profile. I dont understand why its done, what is the need to delete node_modules folder for mvn clean? I dont think we should do it

[gmarziou]

I agree, is it the same for gradle?

[deepu105]

Gradle doesnt do this

Thanks & Regards,
Deepu

On Thu, Apr 7, 2016 at 2:22 PM, Gaël Marziou notifications@github.com
wrote:

I agree, is it the same for gradle?

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#3356 (comment)

[ddewaele]

The new plugin IMHO can be fully backwards compatible with what you currently have.

The only issue I have with the current setup is that it requires software (npm / node) to be available on the system and that it is not controlled by the pom.xml

@deepu105 There can be argumants for and against cleaning node_modules in a prod profile.

• If you don't delete it, and let your CI do an npm install on an existing node_modules over and over, existing modules with a semver will never get updated.
• Depending on how your CI is setup, some nodes might build the app differently. (if you introduce a new node that does an npm installl on a clean repo it can pull in updates dependencies that the other nodes will not see untill you delete their node_modules
• You could also opt to shrinkwrap your devDependencies to keep them stable.

[ddewaele]

@deepu105 I'm getting the following numbers on a clean install with the yeoman plugin. Are you getting the same results ?

Not deleting node_modules speeds things up a little bit, but still majority of the time is lost in the npm install step) :

Keeping node_modules around speeds things up considerably.

mvn command	time	remarks
mvn clean install	00:34 min	Here node_modules is not used.
mvn -Pprod clean install	07:54 min	Using yeoman plugin, without a node_modules present
mvn -Pprod clean install	05:13 min 01:20 min	Using yeoman plugin, with an existing node_modules present and not cleaning it

[deepu105]

@ddewaele IMO if in CI you want the node_modules to be deleted that should be task in your CI to clean up the folder before running mvn . I don't get your point of semver not updating if there is an existing node_modules folder. IMO its just a waste of time to delete and re download the internet again for builds and given the the npm modules are just a dev dependency in a Jhipster app it doesn't make sense to me at all to download all that

And yes npm install can take some time. But without the plugin if I do an npm install on an existing project it hardly takes a minute, so the problem might be the way these plugins work. we could try to do npm install and bower install using command line triggered from the POM to see if there is a difference.

[gmarziou]

@ddewaele IMHO we don't have these issues with node_modules because we use exact versions for our deps in our package.json, so there should not be semver interpolation.

[gmarziou]

Also if you want you can also run `npm prune{ to remove unneeded deps from node_modules

[ddewaele]

@deepu105 with the semver issue I'm just referring to reproducible builds. An update to gulp for example that gets pulled in via semver could in theory generate a different build. If you keep node_modules around an npm install will never update it. Someone else (a developer, or a different CI node) that starts with clean node_modules will get the updated gulp dependency and could generate a different artifact.

@gmarziou : I did see this in the package.json and I'm not sure if transitive dependencies containing semvers can also alter the build, despite using fixed versions in your package.json for your level-1 dependencies.

"gulp-rev-replace": "^0.4.3",

But this is a side-track discussion and not really part of this plugin feature request.

Also noticed just now that the npm install is considerably faster with the node_modules present, so don't what the issue was with my previous run (I'll update the timings).

@deepu105 If we don't see a serious performance degradation by switching to frontend-maven-plugin that offers more control of node/npm and doesn't rely on external installations perhaps that ons is worth considering over the current yeoman plugin ?

[gmarziou]

"gulp-rev-replace": "^0.4.3"

Someone made a mistake, it should be replaced by "0.4.3" as this our project rule, and it's already fixed on master: https://github.com/jhipster/generator-jhipster/blob/master/generators/client/templates/_package.json#L42

[ddewaele]

@gmarziou : Correct. But that still won't prevent the possibility of updated transitive dependencies (with semvers) getting pulled in I think. For that you would need to npm shrinkwrap your dev dependencies.

[gmarziou]

Correct, but cleaning the node_modules does NOT help either.

[ddewaele]

Correct ... Perhaps

• removing the node_modules from the clean goal in the pom
• introducing a shrinkwrap for the package.json

might also be an interesting feature. (not related to this one).

[ddewaele]

To conclude :

• I see added value, making jHipster less dependent on external installations
• Works with Gulp and Grunt
• I think there is no speed degradation by switching to frontend-maven-plugin. The only added step is the download of node/npm. (usually a one-time operation).
• The changes will be limited to the pom.xml. No other impact that I see.
• We can also change the CI docs (will be a lot simpler now)

                    <plugin>
                        <groupId>com.github.eirslett</groupId>
                        <artifactId>frontend-maven-plugin</artifactId>
                        <version>0.0.27</version>
                        <executions>
                            <execution>
                                <id>install node and npm</id>
                                <goals>
                                    <goal>install-node-and-npm</goal>
                                </goals>
                                <configuration>
                                    <nodeVersion>v5.3.0</nodeVersion>
                                    <npmVersion>3.3.12</npmVersion>
                                </configuration>
                            </execution>
                            <execution>
                                <id>npm install</id>
                                <goals>
                                    <goal>npm</goal>
                                </goals>
                                <configuration>
                                    <arguments>install</arguments>
                                </configuration>
                            </execution>
                            <execution>
                                <id>gulp build</id>
                                <goals>
                                    <goal>gulp</goal>
                                </goals>
                                <configuration>
                                    <arguments>build --no-notification</arguments>
                                </configuration>
                            </execution>
                        </executions>
                    </plugin>

@gmarziou : If you guys see added value I'd be happy to create a pull request.

I can also look at the Grunt file if you want (in a seperate issue).

[deepu105]

We do not have grunt anymore.
I still dont see how downloading node_modules for each build is gonna help,
if a transitive dependency update is gonna change the build output then its
an issue that needs to be fixed IMO and thats gonna happen in either case
with or without node_modules present so there is no real difference here.

And I definitely wouldn't want my node_modules deleted and downloaded again
for each prod build.

I dont have a problem in changing the plugin as long as the behaviour is
similar to current, if node_modules is present it shouldn't download stuff
again

@gmarziou I think we should remove node_modules from the pom clean task,
any idea why it was added? May be we should open a seperate issue for that.
On 7 Apr 2016 20:33, "Davy De Waele" notifications@github.com wrote:

To conclude :

• I see added value, making jHipster less dependent on external
  installations

• Works with Gulp and Grunt

• I think there is no speed degradation by switching to
  frontend-maven-plugin
  https://github.com/eirslett/frontend-maven-plugin. The only added
  step is the download of node/npm. (usually a one-time operation).

• The changes will be limited to the pom.xml. No other impact that I
  see.

• We can also change the CI docs
  http://jhipster.github.io/setting-up-ci-linux/ (will be a lot
  simpler now)

              <plugin>
                  <groupId>com.github.eirslett</groupId>
                  <artifactId>frontend-maven-plugin</artifactId>
                  <version>0.0.27</version>
                  <executions>
                      <execution>
                          <id>install node and npm</id>
                          <goals>
                              <goal>install-node-and-npm</goal>
                          </goals>
                          <configuration>
                              <nodeVersion>v5.3.0</nodeVersion>
                              <npmVersion>3.3.12</npmVersion>
                          </configuration>
                      </execution>
                      <execution>
                          <id>npm install</id>
                          <goals>
                              <goal>npm</goal>
                          </goals>
                          <configuration>
                              <arguments>install</arguments>
                          </configuration>
                      </execution>
                      <execution>
                          <id>gulp build</id>
                          <goals>
                              <goal>gulp</goal>
                          </goals>
                          <configuration>
                              <arguments>build --no-notification</arguments>
                          </configuration>
                      </execution>
                  </executions>
              </plugin>
  

@gmarziou https://github.com/gmarziou : If you guys see added value I'd
be happy to create a pull request.

I can also look at the Grunt file if you want (in a seperate issue).

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#3356 (comment)

[ddewaele]

@deepu105 : we don't need to download node_modules for each build (it's already removed from the clean goal in da89a40 from @gmarziou )

Sorry about the whole node_modules discussion. I was confused by the fact that it was in the pom.xml so I thought it was a deliberate decision to put it there. I'm fine with leaving it.

I'm more interested in removing the dependencies to existing node/npm installations.

[gmarziou]

Looks good but there is yet one thing I don't understand, I can see that it installs node and npm, runs npm install, runs gulp but when does it install bower and run bower install?

[ddewaele]

@gmarziou It can do bower install, but I thought that was part of the jhipster bootstrap ? (executing bower install and checking in the bower dependencies).

I assumed developers run that manually, check in their bower file + bower_components artifacts.

[deepu105]

Ok so I went through this in detail and here is my view

1. I like the way current plugin works, its more flexible and doesnt force anything(like node install, npm install etc)
2. Im OK to change to the new plugin if it can do the same process as the old plugin by default in the same runtime (I cannot accept a build which takes around 7 min)
3. for your use case @ddewaele I suggest adding another profile switch ci which could be used to do the additional steps of installing a specific node and npm version

Im sorry for being difficult but hope you understand. Im thinking from the perspective of our normal majority users who might not be using a CI and who are happy with the current flow
@gmarziou @jdubois what do you guys think?

[ddewaele]

(1) You can configure the new plugin to behave exactly the same as the previous one. I don't think it forces you to do anything.

I don't understand why you think the current one is flexible.

• You can't turn off the bower / npm install in the current plugin.
• For every tool it runs it will do 2 completely seperate build / test runs (where often those can be combined in one shot)

I would really suggest you to take a look at the 2 plugins codebases. That might give you a better idea of the quality of both plugin. The yeoman plugin source is one file so that should be quick.

(2) I think this is the case. By combining gulp build and test in one shot it even runs faster with the new plugin IMHO.
(3) ok with that

[deepu105]

@ddewaele As I said I dont mind which plugin we use, so as said Im ok to use the new one if its better.

when I said flexible I was referring to the way we use it currently in the sense it uses whatever is globally installed and locally used in node_modules. Sorry I should have used a better word than flexible.

[deepu105]

also I hope this plugin will use the npm proxy settings if any??

[jdubois]

I think this looks like a lot of work and discussion, and:

• I just don't have the time for this in the next few weeks -> that's my main issue, I just don't have time!
• I don't understand what we gain, outside of the fact that it's more popular
• I understand what we lose and what we risk, and that doesn't sound very good to me
• As long as it's not broken, there's no need to fix the plugin at the moment...

[ddewaele]

Feel free to close this ... I can't keep on defending this. I did my best to explain why :)

[jdubois]

On my side it's only lack of time - I didn't think this would trigger so
many emails and discussions!!!!
I promise I'll have a close look, but we just released JHipster 3.0 and I
can't do everything at the same time. I'm also doing this on my free time,
so it also depends on my "real" job and personnal life, so that's really
nothing against your proposal.
Le 9 avr. 2016 5:30 PM, "Davy De Waele" notifications@github.com a écrit :

Feel free to close this ... I can't keep on defending this. I did my best
to explain why :)

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#3356 (comment)

[atomfrede]

I very long discussion, so I try to give my 2 cents on this. When we started a new project at work I've introduced grunt adn bower as frontend tools together with the yeoman maven plugin we are using (because I knew it from jhipster). Soon we migrated to the frontend maven plugin, for two reasons:

• CI with Jenkins (in a corporate setting you can't always install what you want)
• Collegues not developing frontend

With the frontend maven plugin everyone is able to at least build the war file, deploy it and use the frontend. As we have put all frontend dependencies into target folder doing a clean package takes quite long, so we have added a jenkins profile such that you don't need to download node, npm etc. each time, even if you have it locally installed on your machine. So sounds pretty similar to what @ddewaele and @deepu105 proposed.

For the gradle part, we are already using the mentioned node plugin, so adding the functionality should be no problem.

[deepu105]

@atomfrede thanks man. Its in line with what I wanted as well. So @ddewaele if you could amend it as per mine and Fred's suggestion Ill merge this.

Sorry that its dragging.

@jdubois no worries ill take care of this

[ddewaele]

@atomfrede : About this : "As we have put all frontend dependencies into target folder doing a clean package takes quite long"

This is specific to your project right and not jHipster ?

[atomfrede]

Yes. Some are using eclipse and eclipse still can't handle Javascript and
node very well,therefore we put it into target auch that it is ignore by
default. Jhipster doesn't do that.
Am 09.04.2016 7:50 nachm. schrieb "Davy De Waele" <notifications@github.com

:

@atomfrede https://github.com/atomfrede : About this : "As we have put
all frontend dependencies into target folder doing a clean package takes
quite long"

This is specific to your project right and not jHipster ?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#3356 (comment)

[ddewaele]

@deepu105 : will finish up the PR tomorrow. Thx for all the inputs.

[ddewaele]

I know this has been a long and tiring discussion, but some updates (just want to give as much detail as possible so we can have an objective discussion) :

• The current version of the frontend-maven-plugin will only work if you allow it to install a local version of node/npm.
• It only needs to do this 1 time (a 10MB archive) and unpacks it in ${projectDir}/node. (overhead no more than 10seconds).
• If ${projectDir}/node is kept around if keeps using that and the plugin doesn't need to do anything.
• If ${projectDir}/node is ever deleted it extracts a cached copy from the local maven repo ~/.m2/repository/com/github/eirslett/node/ (so no download is necessary anymore. overhead no more than 3 secs)
• If people have global node/npm/bower installed they can continue using it during their development workflow.
• This eliminates the need for the special CI profile.

So the impact on the build time will be almost non-existing. Only the very first build with -Pprod will you see a 10sec performance penalty. Subsequent builds will be just as fast as before.

The fact you can just drop this in jenkins and treat this like a pure maven project without needing to set anything up remains the strong selling point for this plugin.

The plugin author also states: Not meant to replace the developer version of Node - frontend developers will still install Node on their laptops, but backend developers can run a clean build without even installing Node on their computer.

Still think this is a perfect fit, even with the local node install.

[atomfrede]

@ddewaele Great summary. From my point of view using the plugin is fine (as I said we are using it for 6 month now without big problems). Have you looked the the node gradle plugin? I think it supports the same feature set, so we should make gradle equivalent to maven regarding node handling.

[ddewaele]

I'm not a big gradle fan/user but I'll take a look at it. Shouldn't be that hard.

[atomfrede]

No problem, I can have a look!

[notflorian]

@ddewaele Thanks for your updates, you have convinced me to give it a try on my personal project. From a backend dev point of view, I think it's indeed very nice not to have to install npm globally just to to build the project. 👍

[deepu105]

Im ok if others are ok with this.

On Wed, 13 Apr 2016 5:10 am Florian Fauvarque, notifications@github.com
wrote:

@ddewaele https://github.com/ddewaele Thanks for your updates, you have
convinced me to give it a try on my personal project. From a backend dev
point of view, I think it's indeed very nice not to have to install npm
globally just to to build the project. [image: 👍]

—
You are receiving this because you were mentioned.

Reply to this email directly or view it on GitHub
#3356 (comment)

[deepu105]

closed via #3365

[jdubois]

Ping @eirslett : it took some time but we did it!!!

[eirslett]

Great! 😄