CI: split stages in categories

[v1v]

What does this PR do?

Split the pipeline in four meta-stages:

1. the linting -> requires workers for each merge commit (branches)
2. the build/test for the core platforms (linux, windows-2019 and arm) -> requires workers for each merge commit (branches)
3. the build/test for the extended platforms (windows <=2016, macOS) -> requires workers for each merge commit (branches)
4. the packaging -> requires workers for each merge commit (branches)

Why is it important?

Potentially, this will reduce the CI load by provisioning less number of workers in parallel, which at the moment is causing issues with timeouts. This should help to smooth the required CI workers in batches and affecting a bit less the other pipelines.

Issues

Superseedes #24790

UI

If failures in one of the stages in the build/test meta-stage

If no failures but caching the linting:

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: v1v commented: /test

• Start Time: 2021-04-21T18:01:00.012+0000

• Duration: 95 min 10 sec

• Commit: ea302e2

Test stats 🧪

Test	Results
Failed	0
Passed	32331
Skipped	4323
Total	36654

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	32331
Skipped	4323
Total	36654

[v1v]

/test

[v1v]

/test

[v1v]

/test

[v1v]

/test

[v1v]

/test

[v1v]

/test

[urso]

Why does the lint phase take that many workers?

[v1v]

Why does the lint phase take that many workers?

The linting was split to run specifically per beat, therefore there are 21 lint stages in total defined in each Jenkinsfile.yml. See https://github.com/elastic/beats/search?l=YAML&q=%22Lint%22

This is not something new but something that was added last year -> #22103

We could move away from this approach and use the previous behaviour, aka make check, in favour of using the 1 worker instead of 21.

What do you think?

[kuisathaverat]

the old checkout takes 15 min always, launch the lint in parallel will launch 21 workers only if you modify libbeat or something build everything is the worst case, fo one beat will launch 1-2 and take 5 min or less, I think that worth the quick feedback on linting.

[v1v]

/test

[urso]

the old checkout takes 15 min always, launch the lint in parallel will launch 21 workers only if you modify libbeat or something build everything is the worst case, fo one beat will launch 1-2 and take 5 min or less, I think that worth the quick feedback on linting.

Let's keep it for now then. We should keep this one in mind, in case we need to 'optimize' more.

Hm, this makes me wonder if:

1. we can make the lint step more selective. Do we need to lint Filebeat, even if the change is in libbeat? I would consider that all lints are 'local' to the actual product
2. we can improve make check itself to only run the checks on a preconfigured set of sub-projects (based on environment variable the selective testing support defines).

[v1v]

I raised #25257 to keep track of the previous conversation