[WIP] jenkins: add a node-ci pipeline for running tests

[gibfahn]

Should hopefully be able to replace node-test-pull-request. In future it might also replace node-test-commit if it's possible to easily subsume it's features. Whether we should end up with one giant pipeline containing all the test jobs is a question for later.

Current progress is mostly documented in the pipeline (in comments).

Note that scripted pipelines have to run on master, but for anything that might be more intensive you can just run a custom script on an executor (see the commented out bit in this script).

For the difference between scripted and declarative pipelines, see this.

Progress:

I haven't tested anything after the start of the node-test-commit run yet, I assume it will crash immediately after that. Seems to be working okay.

Latest run: https://ci.nodejs.org/job/node-ci/16/console

Benefits vs node-test-pull-request:

• Nothing in the job other than the repo/branch to pull the pipeline from.
• Can auto-edit the Build description to include useful info like the PR we're testing.

TBD if we should keep this name (to make sure we update everything) or just rename this to node-test-pull-request if it gets good enough.

[joaocgreis]

Just one note for now: it might not be straightforward to replace node-test-commit. Not only because it's used by the daily jobs, but other jobs invoked by it are also used by other jobs (at least the Windows compile and test jobs are used by ChakraCore).

[gibfahn]

Just one note for now: it might not be straightforward to replace node-test-commit

Makes sense. If I get node-test-pull-request working I'll land this and we can iterate, I'd like to make changes incrementally as much as possible 😁 .

A path forward might be to modify this job to do the same thing node-test-commit does (in a future PR), and then slowly move over other things using it while keeping node-test-commit around. Then when we think everything is done we can try disabling it (temporarily) and see what breaks.

Or maybe it should continue to be a separate job, IDK.

[rvagg]

This demonstrates my critique of pipelines pretty well - how are our users going to adapt to this? They have to go into the console to find a link to node-test-commit, and the console is full of useless and confusing messages so you have to filter through the blue links. I thought blue-ocean was meant to make these things easier? Is there a way to make this less opaque to the user who doesn't care whether we are using pipelines or traditional jobs?

[gibfahn]

This demonstrates my critique of pipelines pretty well - how are our users going to adapt to this?

Good question

They have to go into the console to find a link to node-test-commit

I'd say that's not the end of the world, although a less useless status view of the pipeline would be really useful.

the console is full of useless and confusing messages so you have to filter through the blue links.

Most of that is debugging info and can be removed. The irrelevant text is greyed out, so it's pretty easy to find the blue links right?

This is the minimal console output you can get (AFAIK).

I thought blue-ocean was meant to make these things easier?

In my experience the only thing blue ocean offers is the ability to show multi-config jobs (i.e. the picture below) and it is currently unbelievably slow to load, even with a super-simple job like this. Apparently there's a bug on their roadmap.

Is there a way to make this less opaque to the user who doesn't care whether we are using pipelines or traditional jobs?

This is definitely a good question, and we should have a look at whether there are any solutions out there. To be honest for me the benefits of pipelines outweigh that cost, but there may well be other costs we haven't hit yet that will make them infeasible.

Also the user shouldn't really be looking at node-test-commit, they should be looking at the tickbox on the PR itself. The fact that they can't currently rely on that is a problem to be solved IMO.

[gibfahn]

This is the minimal console output FWIW:

Maybe pipelines aren't the answer, but the alternative is a series of build plugins that will presumably be even less maintained going forward. Maybe the answer is to work with the Jenkins team to fix the issues we have with pipelines.

[gibfahn]

Actually thinking about it further, if the Blue Ocean bug was fixed, and it's on their roadmap then I think that would resolve the issue.

[sam-github]

Our current plan is not to use pipelines, as I understand from recent discussion, the pipeline experiment was not found to be worth continuing. Closing, but it can be reopened if we ever decide to give pipelines a try again.