[CI] Pipeline 2.0 for monorepos #20104

v1v · 2020-07-21T16:24:48Z

What does this PR do?

This is the very attempt for the Pipeline 2.0 for monorepos that should help us to:

easy to maintain/collaborate
an abstraction to simplify what each beat does
enhance what are the actions that should trigger a build.

It should dynamically create the stages accordingly and also a clear markdown document that explains what are the reasons for that particular build to trigger those stages.

Why is it important?

Being able to scale and new features to the pipeline.

Use Cases

The below list of use cases should help to understand what are the scenarios that we would like to cover with this implementation.

Id	Description	Status
1	As a contributor/reviewer to the auditbeat project I want my PR to be verified with the CI for all the mandatory stages.	✅
2	As a reviewer to the auditbeat project I want the PR that I review to be validated with the CI for all the optionals stages that I did configure it with a github label / a comment / a manual build by using the UI.	✅
3	As a contributor to the auditbeat project I want my PR to be verified for macOS or Window when required	✅
4	As a contributor to the auditbeat project I want my PR to be skipped in the CI when I configured it with a github label.	❔
5	As a contributor to the libbeats project I want my PR to verify all the other beats with their mandatory stages.	✅
6	As a contributor to the auditbeat project I want the build report for my PR to describe what it happened, why it happened and how to reproduce it locally.	✅
7	As a contributor to the auditbeat project I want my draft PR to be skipped in the CI.	❔
8	As a branch/tag I want the build to run for all the stages unless it's set to false.	✅
9	As a parenstream job I want to trigger a downstream build.	🔢

Further details

See Pipeline As Yaml as they suggest certain syntax that could help, maybe, to align with it ?!

Tasks

testGCPFunctions
Enable test goals in some projects as soon as the tests are fine.
Dump Environment variables
getProjectDependencies
AWS tests. This is failing even in the 1.0 -> [x-pack/metricbeat] ec2_integration_test #20951

Build stages

The below list has been generated with the script .ci/scripts/generate_build_table.py | pbcopy

It should replace the excel with the data we have been building so far

Beat	Stage	Command	MODULE	Platforms	When
auditbeat	build	`mage build test`	False	`['linux && ubuntu-16']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:
auditbeat	crosscompile	`make -C auditbeat crosscompile`	False	`['linux && ubuntu-16']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:
auditbeat	macos	`mage build unitTest`	False	`['macosx']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:/:star:
auditbeat	windows	`mage build unitTest`	False	`['windows-2019', 'windows-2016']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:/:star:
deploy/kubernetes	k8sTest	`mage build unitTest`	False	`['linux && ubuntu-18']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:
filebeat	build	`mage build test`	True	`['linux && ubuntu-18']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:
filebeat	macos	`mage build unitTest`	False	`['macosx']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:/:star:
filebeat	windows	`mage build unitTest`	False	`['windows-2019']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:/:star:
generator	metricbeat-test	`make -C generator/_templates/metricbeat test test-package`	False	`['linux && ubuntu-18']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:
generator	beat-test	`make -C generator/_templates/beat test test-package`	False	`['linux && ubuntu-18']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:
generator	macos-metricbeat	`make -C generator/_templates/metricbeat test`	False	`['macosx']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:/:star:
generator	macos-beat	`make -C generator/_templates/beat test`	False	`['macosx']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:/:star:/:star:
heartbeat	build	`mage build test`	False	`['linux && ubuntu-18']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:
heartbeat	macos	`mage build unitTest`	False	`['macosx']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:/:star:
heartbeat	windows	`mage build unitTest`	False	`['windows-2019']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:/:star:
journalbeat	unitTest	`mage build unitTest`	False	`['linux && ubuntu-18']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:
libbeat	build	`mage build test`	False	`['linux && ubuntu-18']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:
libbeat	crosscompile	`make -C libbeat crosscompile`	False	`['linux && ubuntu-18']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:
libbeat	stress-tests	`make STRESS_TEST_OPTIONS='-timeout=20m -race -v -parallel 1' -C libbeat stress-tests`	False	`['linux && ubuntu-18']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:
metricbeat	unitTest	`mage build unitTest`	False	`['linux && ubuntu-18']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:
metricbeat	goIntegTest	`mage goIntegTest`	True	`['linux && ubuntu-18']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:
metricbeat	pythonIntegTest	`mage pythonIntegTest`	True	`['linux && ubuntu-18']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:
metricbeat	crosscompile	`make -C metricbeat crosscompile`	False	`['linux && ubuntu-18']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:
metricbeat	macos	`mage build unitTest`	False	`['macosx']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:/:star:
metricbeat	windows	`mage build unitTest`	False	`['windows-2019']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:/:star:
packetbeat	build	`mage build test`	False	`['linux && ubuntu-18']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:
packetbeat	macos	`mage build unitTest`	False	`['macosx']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:/:star:
packetbeat	windows	`mage build unitTest`	False	`['windows-2019']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:/:star:
winlogbeat	crosscompile	`make -C winlogbeat crosscompile`	False	`['linux && ubuntu-18']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:
winlogbeat	windows	`mage build unitTest`	False	`['windows-2019']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:
x-pack/auditbeat	build	`mage update build test`	True	`['linux && ubuntu-18']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:
x-pack/auditbeat	macos	`mage build unitTest`	False	`['macosx']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:/:star:
x-pack/auditbeat	windows	`mage build unitTest`	False	`['windows-2019', 'windows-2016']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:/:star:
x-pack/dockerlogbeat	build	`mage build test`	True	`['linux && ubuntu-18']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:
x-pack/elastic-agent	build	`mage build test`	False	`['linux && ubuntu-18']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:
x-pack/elastic-agent	macos	`mage build unitTest`	False	`['macosx']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:/:star:
x-pack/elastic-agent	windows	`mage build unitTest`	False	`['windows-2019']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:/:star:
x-pack/filebeat	build	`mage build test`	True	`['linux && ubuntu-18']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:
x-pack/filebeat	macos	`mage build unitTest`	False	`['macosx']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:/:star:
x-pack/filebeat	windows	`mage build unitTest`	False	`['windows-2019']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:/:star:
x-pack/functionbeat	build	`mage build test && GO_VERSION=1.13.1 mage testGCPFunctions`	False	`['linux && ubuntu-18']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:
x-pack/functionbeat	macos	`mage build unitTest`	False	`['macosx']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:/:star:
x-pack/functionbeat	windows	`mage build unitTest`	False	`['windows-2019']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:/:star:
x-pack/libbeat	build	`mage build test`	False	`['linux && ubuntu-18']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:
x-pack/metricbeat	build	`mage build test`	True	`['linux && ubuntu-18']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:/:star:
x-pack/metricbeat	macos	`mage build unitTest`	False	`['macosx']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:/:star:/:star:
x-pack/metricbeat	windows	`mage build unitTest`	False	`['windows-2019']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:/:star:/:star:
x-pack/packetbeat	windows	`mage build unitTest`	True	`['windows-2019']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:
x-pack/winlogbeat	build	`mage build unitTest`	True	`['windows-2019']`	/:palm_tree:/:file_folder:/:speech_balloon:/:label:/:smiley:/:taco:

🌴 -> Git Branch based
🏷️ -> GitHub Pull Request Label based
📁 -> Changeset based
💬 -> GitHub Pull Request comment based
🌮 -> Git tag based
😃 -> Manual UI interaction based
⭐ -> More specific cases based

Follow ups

Build reason started comment, similar approach as the markdown generated automatically but more specifically about what stages and the reason. TBD in a follow up

elasticmachine · 2020-07-21T17:24:45Z

💔 Tests Failed

Expand to view the summary

Build stats

Build Cause: [Pull request #20104 updated]
Start Time: 2020-09-24T15:18:23.171+0000
Duration: 75 min 17 sec

Test stats 🧪

Test	Results
Failed	2
Passed	20170
Skipped	1854
Total	22026

Test errors

Expand to view the tests failures

Name: Build&Test / heartbeat-macos-macosx / test_config_add – heartbeat.tests.system.test_reload.Test
- Age: 1
- Duration: 10.227
- Error Details: beat.beat.TimeoutError: Timeout waiting for 'cond' to be true. Waited 10 seconds.
Name: Build&Test / libbeat-build / TestClientPublishEventKerberosAware – elasticsearch
- Age: 5
- Duration: 2.51
- Error Details: Failed

Steps errors

Expand to view the steps failures

Name: mage build unitTest
- Description: mage build unitTest
- Duration: 4 min 32 sec
- Start Time: 2020-09-24T16:19:03.182+0000
- log
Name: Notifies GitHub of the status of a Pull Request
- Description: script returned exit code 1
- Duration: 0 min 1 sec
- Start Time: 2020-09-24T16:22:47.727+0000
- log
Name: mage build test
- Description: mage build test
- Duration: 20 min 42 sec
- Start Time: 2020-09-24T15:43:45.002+0000
- log
Name: Notifies GitHub of the status of a Pull Request
- Description: script returned exit code 1
- Duration: 0 min 1 sec
- Start Time: 2020-09-24T16:03:43.560+0000
- log

Log output

Expand to view the last 100 lines of log output

[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/journalbeat-unitTest/journalbeat/build/TEST-python-unit.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/journalbeat-unitTest/journalbeat/build/TEST-go-unit.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/auditbeat-build/auditbeat/build/TEST-python-unit.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/auditbeat-build/auditbeat/build/TEST-python-integration.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/auditbeat-build/auditbeat/build/TEST-go-integration.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/auditbeat-build/auditbeat/build/TEST-go-unit.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-metricbeat-macos-macosx/x-pack/metricbeat/build/TEST-python-unit.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-metricbeat-macos-macosx/x-pack/metricbeat/build/TEST-go-unit.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-stan.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-sql.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-azure.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-googlecloud.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-cloudfoundry.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-python-unit.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-ibmmq.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-python-integration.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-prometheus.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-cockroachdb.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-statsd.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-aws.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-oracle.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-coredns.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-mssql.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-appsearch.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-redisenterprise.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-unit.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-dockerlogbeat-build/x-pack/dockerlogbeat/build/TEST-go-unit.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/filebeat-build/filebeat/build/TEST-python-unit.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/filebeat-build/filebeat/build/TEST-python-integration.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/filebeat-build/filebeat/build/TEST-go-integration.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/filebeat-build/filebeat/build/TEST-go-unit.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/heartbeat-macos-macosx/heartbeat/build/TEST-python-unit.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/heartbeat-macos-macosx/heartbeat/build/TEST-go-unit.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-filebeat-windows-windows-2019/x-pack/filebeat/build/TEST-python-unit.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-filebeat-windows-windows-2019/x-pack/filebeat/build/TEST-go-unit.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/metricbeat-unitTest/metricbeat/build/TEST-python-unit.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/metricbeat-unitTest/metricbeat/build/TEST-go-unit.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-functionbeat-windows-windows-2019/x-pack/functionbeat/build/TEST-python-unit.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-functionbeat-windows-windows-2019/x-pack/functionbeat/build/TEST-go-unit.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-functionbeat-build/x-pack/functionbeat/build/TEST-python-unit.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/x-pack-functionbeat-build/x-pack/functionbeat/build/TEST-go-unit.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/metricbeat-pythonIntegTest/metricbeat/build/TEST-python-integration.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/auditbeat-windows-windows-2019/auditbeat/build/TEST-python-unit.xml
[2020-09-24T16:33:17.244Z] ./src/github.com/elastic/beats/auditbeat-windows-windows-2019/auditbeat/build/TEST-go-unit.xml
[2020-09-24T16:33:17.244Z] + cat
[2020-09-24T16:33:17.244Z] + /usr/local/bin/runbld ./runbld-script --job-name elastic+beats+pull-request
[2020-09-24T16:33:17.244Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-09-24T16:33:23.831Z] runbld>>> runbld started
[2020-09-24T16:33:23.831Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-09-24T16:33:24.776Z] runbld>>> The following profiles matched the job 'elastic+beats+pull-request' in order of occurrence in the config (last value wins).
[2020-09-24T16:33:24.776Z] runbld>>> Matches in the system config:
[2020-09-24T16:33:24.776Z] runbld>>> - Matched ^elastic\+beats
[2020-09-24T16:33:24.776Z] runbld>>> - Matched ^elastic\+beats\+pull-request
[2020-09-24T16:33:26.160Z] runbld>>> Debug logging enabled.
[2020-09-24T16:33:26.160Z] runbld>>> Storing result
[2020-09-24T16:33:26.421Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-09-24T16:33:26.421Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20200924163326-0C840A92
[2020-09-24T16:33:26.421Z] runbld>>> Adding system facts.
[2020-09-24T16:33:27.803Z] runbld>>> Adding vcs info for the latest commit:  778c037f5518c66e4ca80c3761d0f700e1bf0169
[2020-09-24T16:33:27.803Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-09-24T16:33:27.803Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-09-24T16:33:27.803Z] + echo 'Processing JUnit reports with runbld...'
[2020-09-24T16:33:27.803Z] Processing JUnit reports with runbld...
[2020-09-24T16:33:28.062Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-09-24T16:33:28.062Z] runbld>>> DURATION: 24ms
[2020-09-24T16:33:28.062Z] runbld>>> STDOUT: 40 bytes
[2020-09-24T16:33:28.062Z] runbld>>> STDERR: 49 bytes
[2020-09-24T16:33:28.062Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-09-24T16:33:28.062Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-20104
[2020-09-24T16:33:29.000Z] runbld>>> Storing build metadata: 
[2020-09-24T16:33:29.000Z] runbld>>> Adding test report.
[2020-09-24T16:33:29.000Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-20104/src/github.com/elastic/beats
[2020-09-24T16:33:29.568Z] runbld>>> Found 141 test output files
[2020-09-24T16:33:30.135Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20104/src/github.com/elastic/beats/metricbeat-goIntegTest/metricbeat/build/TEST-go-integration-graphite.xml
[2020-09-24T16:33:30.135Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20104/src/github.com/elastic/beats/metricbeat-goIntegTest/metricbeat/build/TEST-go-integration-windows.xml
[2020-09-24T16:33:32.670Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20104/src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-09-24T16:33:32.670Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20104/src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-09-24T16:33:32.670Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20104/src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-09-24T16:33:32.670Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20104/src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-09-24T16:33:32.670Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20104/src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-09-24T16:33:33.607Z] runbld>>> Test output logs contained: Errors: 0 Failures: 2 Tests: 22026 Skipped: 1561
[2020-09-24T16:33:33.866Z] runbld>>> Storing result
[2020-09-24T16:33:33.866Z] runbld>>> FAILURES: 2
[2020-09-24T16:33:34.434Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-09-24T16:33:34.434Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20200924163326-0C840A92
[2020-09-24T16:33:34.434Z] runbld>>> Email notification disabled by environment variable.
[2020-09-24T16:33:34.434Z] runbld>>> Slack notification disabled by environment variable.
[2020-09-24T16:33:39.879Z] Running on worker-1244230 in /var/lib/jenkins/workspace/Beats_beats_PR-20104
[2020-09-24T16:33:39.911Z] [INFO] getVaultSecret: Getting secrets
[2020-09-24T16:33:40.005Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-09-24T16:33:42.012Z] + chmod 755 generate-build-data.sh
[2020-09-24T16:33:42.012Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20104/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20104/runs/64 FAILURE 4517447
[2020-09-24T16:33:42.012Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20104/runs/64/steps/?limit=10000 -o steps-info.json
[2020-09-24T16:33:49.737Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20104/runs/64/tests/?status=FAILED -o tests-errors.json
[2020-09-24T16:33:50.437Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20104/runs/64/log/ -o pipeline-log.txt

auditbeat/Jenkinsfile.yml

Co-authored-by: Ivan Fernandez Calvo <kuisathaverat@users.noreply.github.com>

…ne-2.0 * upstream/master: (41 commits) adding possibility to override content-type checks, it was breaking certain webhooks that is not able to set content-headers at all. Still defaults to application/json (elastic#20232) fix: use a fixed worker type for tests (elastic#20130) [Ingest Manager] Prepare packaging for endpoint and asc files (elastic#20186) [Packetbeat] HTTP: Improve support for 100-continue elastic#15830 (elastic#19349) Increase index.max_docvalue_fields_search to 200 (elastic#20218) [Ingest Manager] Prevent closing closed reader (elastic#20214) [Metricbeat] Use MySQL Host Parser in Query metricset (elastic#20191) Testing: Ignore timestamp from cylance/protect dataset (elastic#20211) [Filebeat] Ignore cylance.protect timestamps while testing (elastic#20207) [CI] remove codecov step (elastic#20102) [docs] Indicate that SYSTEM user is required on Windows to use Endpoint (elastic#20172) Remove f5/firepass rsa2elk fileset (elastic#20160) [Elastic Agent] Improve GRPC stop to be more relaxed. (elastic#20118) Fix fileset field prefixing (elastic#20170) Fix terminating pod autodiscover issue (elastic#20084) Call host parser only once when building light metricsets (elastic#20149) [CI] fix null string with contains (elastic#20182) [Ingest Manager] Fix failing unit tests on windows (elastic#20127) [Filebeat] Update crowdstrike module (elastic#20138) [docs] Add x-pack role to relevant metricsets (elastic#20167) ...

…eature/ci-pipeline-2.0 * 'feature/ci-pipeline-2.0' of github.com:v1v/beats: Update auditbeat/Jenkinsfile.yml Update auditbeat/Jenkinsfile.yml

.ci/Jenkinsfile

This reverts commit 56c45d5.

…ne-2.0 * upstream/master: (33 commits) Stop running agent container as root by default (elastic#21213) Stop running auditbeat container as root by default (elastic#21202) Fix autodiscover flaky tests (elastic#21242) [Ingest Manager] Enabled dev builds (elastic#21241) Fix librpm installation in auditbeat build (elastic#21239) Fix prometheus default config (elastic#21253) Fix dev guide test command (elastic#21254) Move aws lambda metricset to GA (elastic#21255) [Docs] Typo in table syntax (elastic#20227) [ECS] Adds related.hosts to capture all hostnames and host identifiers on an event. (elastic#21160) Add recursive split to httpjson (elastic#21214) [DOCS] Add beat specific start widgets (elastic#21217) Fix timestamp handling in remote_write (elastic#21166) Fix aws, azure and googlecloud compute dashboards (elastic#21098) Add acceptable event log keys to winlog (elastic#21205) Add elastic-agent to gitignore (elastic#21219) Add cloudfoundry tags to events (elastic#21177) [Ingest Manager] Agent includes pgp file (elastic#19480) Add compatibility note about ingress-controller-v0.34.1 (elastic#21209) [Ingest Manager] Support for UPGRADE_ACTION (elastic#21002) ...

…ne-2.0-arm * upstream/master: [Ingest manager] Copy Action store on upgrade (elastic#21298) [CI] Pipeline 2.0 for monorepos (elastic#20104) Stop running agent container as root by default (elastic#21213) Stop running auditbeat container as root by default (elastic#21202) Fix autodiscover flaky tests (elastic#21242) [Ingest Manager] Enabled dev builds (elastic#21241)

…-refactor * upstream/master: [Ingest manager] Copy Action store on upgrade (elastic#21298) [CI] Pipeline 2.0 for monorepos (elastic#20104) Stop running agent container as root by default (elastic#21213) Stop running auditbeat container as root by default (elastic#21202) Fix autodiscover flaky tests (elastic#21242) [Ingest Manager] Enabled dev builds (elastic#21241) Fix librpm installation in auditbeat build (elastic#21239) Fix prometheus default config (elastic#21253) Fix dev guide test command (elastic#21254) Move aws lambda metricset to GA (elastic#21255) [Docs] Typo in table syntax (elastic#20227) [ECS] Adds related.hosts to capture all hostnames and host identifiers on an event. (elastic#21160)

* upstream/master: (417 commits) libbeat/cmd/instance: report cgroup stats (elastic#21113) Configurable index template loading (elastic#21212) [Ingest Manager] Thread safe sorted set (elastic#21290) Change mirror of kafka download (elastic#19645) [Ingest manager] Copy Action store on upgrade (elastic#21298) [CI] Pipeline 2.0 for monorepos (elastic#20104) Stop running agent container as root by default (elastic#21213) Stop running auditbeat container as root by default (elastic#21202) Fix autodiscover flaky tests (elastic#21242) [Ingest Manager] Enabled dev builds (elastic#21241) Fix librpm installation in auditbeat build (elastic#21239) Fix prometheus default config (elastic#21253) Fix dev guide test command (elastic#21254) Move aws lambda metricset to GA (elastic#21255) [Docs] Typo in table syntax (elastic#20227) [ECS] Adds related.hosts to capture all hostnames and host identifiers on an event. (elastic#21160) Add recursive split to httpjson (elastic#21214) [DOCS] Add beat specific start widgets (elastic#21217) Fix timestamp handling in remote_write (elastic#21166) Fix aws, azure and googlecloud compute dashboards (elastic#21098) ...

* upstream/master: (399 commits) libbeat/cmd/instance: report cgroup stats (elastic#21113) Configurable index template loading (elastic#21212) [Ingest Manager] Thread safe sorted set (elastic#21290) Change mirror of kafka download (elastic#19645) [Ingest manager] Copy Action store on upgrade (elastic#21298) [CI] Pipeline 2.0 for monorepos (elastic#20104) Stop running agent container as root by default (elastic#21213) Stop running auditbeat container as root by default (elastic#21202) Fix autodiscover flaky tests (elastic#21242) [Ingest Manager] Enabled dev builds (elastic#21241) Fix librpm installation in auditbeat build (elastic#21239) Fix prometheus default config (elastic#21253) Fix dev guide test command (elastic#21254) Move aws lambda metricset to GA (elastic#21255) [Docs] Typo in table syntax (elastic#20227) [ECS] Adds related.hosts to capture all hostnames and host identifiers on an event. (elastic#21160) Add recursive split to httpjson (elastic#21214) [DOCS] Add beat specific start widgets (elastic#21217) Fix timestamp handling in remote_write (elastic#21166) Fix aws, azure and googlecloud compute dashboards (elastic#21098) ...

* upstream/master: (60 commits) libbeat/cmd/instance: report cgroup stats (elastic#21113) Configurable index template loading (elastic#21212) [Ingest Manager] Thread safe sorted set (elastic#21290) Change mirror of kafka download (elastic#19645) [Ingest manager] Copy Action store on upgrade (elastic#21298) [CI] Pipeline 2.0 for monorepos (elastic#20104) Stop running agent container as root by default (elastic#21213) Stop running auditbeat container as root by default (elastic#21202) Fix autodiscover flaky tests (elastic#21242) [Ingest Manager] Enabled dev builds (elastic#21241) Fix librpm installation in auditbeat build (elastic#21239) Fix prometheus default config (elastic#21253) Fix dev guide test command (elastic#21254) Move aws lambda metricset to GA (elastic#21255) [Docs] Typo in table syntax (elastic#20227) [ECS] Adds related.hosts to capture all hostnames and host identifiers on an event. (elastic#21160) Add recursive split to httpjson (elastic#21214) [DOCS] Add beat specific start widgets (elastic#21217) Fix timestamp handling in remote_write (elastic#21166) Fix aws, azure and googlecloud compute dashboards (elastic#21098) ...

This commit introduces a main pipeline generator similar to the one used with Jenkins #20104 to help with the migration to Buildkite. Co-authored-by: Pavel Zorin <pavel.zorin@elastic.co> Co-authored-by: Alexandros Sapranidis <alexandros@elastic.co>

This commit introduces a main pipeline generator similar to the one used with Jenkins #20104 to help with the migration to Buildkite. Co-authored-by: Pavel Zorin <pavel.zorin@elastic.co> Co-authored-by: Alexandros Sapranidis <alexandros@elastic.co> (cherry picked from commit 4b7467b)

This commit introduces a main pipeline generator similar to the one used with Jenkins #20104 to help with the migration to Buildkite. Co-authored-by: Pavel Zorin <pavel.zorin@elastic.co> Co-authored-by: Alexandros Sapranidis <alexandros@elastic.co> (cherry picked from commit 4b7467b) # Conflicts: # .buildkite/hooks/pre-command

This commit introduces a main pipeline generator similar to the one used with Jenkins #20104 to help with the migration to Buildkite. Co-authored-by: Pavel Zorin <pavel.zorin@elastic.co> Co-authored-by: Alexandros Sapranidis <alexandros@elastic.co> (cherry picked from commit 4b7467b) # Conflicts: # .buildkite/hooks/pre-command Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

This commit introduces a main pipeline generator similar to the one used with Jenkins #20104 to help with the migration to Buildkite. Co-authored-by: Pavel Zorin <pavel.zorin@elastic.co> Co-authored-by: Alexandros Sapranidis <alexandros@elastic.co> (cherry picked from commit 4b7467b) Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

[CI] Pipeline 2.0 for monorepos

ac465eb

v1v self-assigned this Jul 21, 2020

v1v added the automation label Jul 21, 2020

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jul 21, 2020

v1v added ci Team:Automation Label for the Observability productivity team labels Jul 21, 2020

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jul 21, 2020

kuisathaverat reviewed Jul 22, 2020

View reviewed changes

auditbeat/Jenkinsfile.yml Outdated Show resolved Hide resolved

kuisathaverat reviewed Jul 22, 2020

View reviewed changes

auditbeat/Jenkinsfile.yml Outdated Show resolved Hide resolved

v1v and others added 5 commits July 23, 2020 16:38

Update auditbeat/Jenkinsfile.yml

aa6a07a

Co-authored-by: Ivan Fernandez Calvo <kuisathaverat@users.noreply.github.com>

Update auditbeat/Jenkinsfile.yml

1b91c12

Co-authored-by: Ivan Fernandez Calvo <kuisathaverat@users.noreply.github.com>

[CI] 2.0 YAML to dynamic stage generations

7183f93

Merge branch 'feature/ci-pipeline-2.0' of github.com:v1v/beats into f…

4290714

…eature/ci-pipeline-2.0 * 'feature/ci-pipeline-2.0' of github.com:v1v/beats: Update auditbeat/Jenkinsfile.yml Update auditbeat/Jenkinsfile.yml

kuisathaverat reviewed Jul 27, 2020

View reviewed changes

.ci/Jenkinsfile Outdated Show resolved Hide resolved

Use shared library

ca9a5a6

v1v mentioned this pull request Jul 28, 2020

2.0 pipeline steps for beats elastic/apm-pipeline-library#689

Merged

3 tasks

v1v added 10 commits July 28, 2020 14:20

Use beatsWhen and beatsStages

56c45d5

Revert "Use beatsWhen and beatsStages"

778da9e

This reverts commit 56c45d5.

Simplify

daf8fb1

Refactor

fcab64b

Add run commands

6a0f1bd

use nodeOS step

a8cc866

Fix GO_VERSION

14e039d

Transform to String

7660a6d

Use method-pointer-operator

35d30be

replace with isInstalled

6be81e2

v1v added the macOS Enable builds in the CI for darwin testing label Jul 28, 2020

Fix the YAML data

e874d54

v1v merged commit aad07ad into elastic:master Sep 24, 2020

v1v deleted the feature/ci-pipeline-2.0 branch September 24, 2020 16:54

v1v mentioned this pull request Sep 24, 2020

Cherry-pick to 7.x: [CI] Pipeline 2.0 for monorepos (#20104) #21312

Merged

v1v added a commit to v1v/beats that referenced this pull request Sep 24, 2020

[CI] Pipeline 2.0 for monorepos (elastic#20104)

bcc9f53

v1v added a commit to v1v/beats that referenced this pull request Sep 24, 2020

[CI] Pipeline 2.0 for monorepos (elastic#20104)

99ccc31

This was referenced Sep 24, 2020

Cherry-pick to 7.9: [CI] Pipeline 2.0 for monorepos (#20104) #21313

Merged

[CI] cherry-pick #20891 #21314

Merged

v1v added a commit that referenced this pull request Sep 28, 2020

[CI] Pipeline 2.0 for monorepos (#20104) (#21312)

e076e9a

v1v added a commit that referenced this pull request Sep 28, 2020

[CI] Pipeline 2.0 for monorepos (#20104) (#21313)

6fc6573

v1v mentioned this pull request Sep 28, 2020

Support skip-ci-build label for beatsWhen elastic/apm-pipeline-library#733

Merged

mdelapenya mentioned this pull request Jul 27, 2021

chore: decouple unit and integration tests on CI #27069

Merged

6 tasks

v1v mentioned this pull request Mar 14, 2022

ci: split windows in mandatory and ext_win stages #30781

Merged

1 task

leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023

[CI] Pipeline 2.0 for monorepos (elastic#20104) (elastic#21313)

190f7d5

v1v mentioned this pull request Mar 4, 2024

PoC: use pipeline generator #38162

Merged

13 tasks

mergify bot mentioned this pull request Mar 20, 2024

[8.13](backport #38162) PoC: use pipeline generator #38478

Merged

13 tasks

mergify bot mentioned this pull request Mar 20, 2024

[7.17](backport #38162) PoC: use pipeline generator #38485

Merged

13 tasks

mergify bot mentioned this pull request Mar 20, 2024

[8.12](backport #38162) PoC: use pipeline generator #38486

Merged

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI] Pipeline 2.0 for monorepos #20104

[CI] Pipeline 2.0 for monorepos #20104

v1v commented Jul 21, 2020 •

edited

Loading

elasticmachine commented Jul 21, 2020 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

[CI] Pipeline 2.0 for monorepos #20104

[CI] Pipeline 2.0 for monorepos #20104

Conversation

v1v commented Jul 21, 2020 • edited Loading

What does this PR do?

Why is it important?

Use Cases

Further details

Tasks

Build stages

Follow ups

elasticmachine commented Jul 21, 2020 • edited by jenkins-beats-ci bot Loading

💔 Tests Failed

Build stats

Test stats 🧪

Test errors

Steps errors

Log output

v1v commented Jul 21, 2020 •

edited

Loading

elasticmachine commented Jul 21, 2020 •

edited by jenkins-beats-ci bot

Loading