Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PoC: split mandatory from other stages #24790

Closed
wants to merge 8 commits into from

Conversation

v1v
Copy link
Member

@v1v v1v commented Mar 26, 2021

What does this PR do?

Split the pipeline in three meta-stages:

  • the first stage - linting
  • the second stage - run the build/test for the core platforms (linux, windows-2019 and arm)
  • the third one to run the build/test for the extended platforms (windows <=2016, macOS) besides running that packaging.

From the user point of view, this could help to discard flakiness and trigger the specific stage:

  • /test <beats> runs the first stage
  • /test ext <beats> runs the second stage.

Why is it important?

Explore a way to improve the flakiness by splitting what's stable versus what's a bit more flaky.

Potentially, this will reduce the CI load by provisioning less number of workers in parallel, which at the moment is causing issues with timeouts. This should help to smooth the required CI workers in batches and affecting a bit less the other pipelines.

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Mar 26, 2021
@elasticmachine
Copy link
Collaborator

elasticmachine commented Mar 26, 2021

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: v1v commented: /test

  • Start Time: 2021-04-15T12:55:17.641+0000

  • Duration: 26 min 58 sec

  • Commit: 0a0faef

Test stats 🧪

Test Results
Failed 1
Passed 2395
Skipped 3
Total 2399

Trends 🧪

Image of Build Times

Image of Tests

Test errors 1

Expand to view the tests failures

TestMonitorSample – github.com/elastic/beats/v7/libbeat/metric/system/cpu
    Expand to view the error details

     Failed 
    

    Expand to view the stacktrace

     === RUN   TestMonitorSample
        cpu_test.go:55: 
            	Error Trace:	cpu_test.go:55
            	Error:      	Should be true
            	Test:       	TestMonitorSample
        cpu_test.go:56: 
            	Error Trace:	cpu_test.go:56
            	Error:      	Should be true
            	Test:       	TestMonitorSample
    --- FAIL: TestMonitorSample (0.00s)
     
    

Steps errors 2

Expand to view the steps failures

libbeat-arm-arm - mage build unitTest
  • Took 9 min 32 sec . View more details on here
  • Description: mage build unitTest
Error signal
  • Took 0 min 0 sec . View more details on here
  • Description: Error 'hudson.AbortException: script returned exit code 1'

Log output

Expand to view the last 100 lines of log output

[2021-04-15T13:20:27.364Z]  Built:             Fri Jan 29 14:33:34 2021
[2021-04-15T13:20:27.364Z]  OS/Arch:           linux/arm64
[2021-04-15T13:20:27.364Z]  Context:           default
[2021-04-15T13:20:27.364Z]  Experimental:      true
[2021-04-15T13:20:27.364Z] 
[2021-04-15T13:20:27.364Z] Server: Docker Engine - Community
[2021-04-15T13:20:27.364Z]  Engine:
[2021-04-15T13:20:27.364Z]   Version:          20.10.3
[2021-04-15T13:20:27.364Z]   API version:      1.41 (minimum version 1.12)
[2021-04-15T13:20:27.364Z]   Go version:       go1.13.15
[2021-04-15T13:20:27.364Z]   Git commit:       46229ca
[2021-04-15T13:20:27.364Z]   Built:            Fri Jan 29 14:31:49 2021
[2021-04-15T13:20:27.364Z]   OS/Arch:          linux/arm64
[2021-04-15T13:20:27.364Z]   Experimental:     false
[2021-04-15T13:20:27.364Z]  containerd:
[2021-04-15T13:20:27.364Z]   Version:          1.4.4
[2021-04-15T13:20:27.364Z]   GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
[2021-04-15T13:20:27.364Z]  runc:
[2021-04-15T13:20:27.364Z]   Version:          1.0.0-rc93
[2021-04-15T13:20:27.364Z]   GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
[2021-04-15T13:20:27.364Z]  docker-init:
[2021-04-15T13:20:27.364Z]   Version:          0.19.0
[2021-04-15T13:20:27.364Z]   GitCommit:        de40ad0
[2021-04-15T13:20:27.364Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-04-15T13:20:27.364Z] Unable to find image 'arm64v8/alpine:3' locally
[2021-04-15T13:20:28.444Z] 3: Pulling from arm64v8/alpine
[2021-04-15T13:20:28.444Z] 595b0fe564bb: Pulling fs layer
[2021-04-15T13:20:28.959Z] 595b0fe564bb: Verifying Checksum
[2021-04-15T13:20:28.960Z] 595b0fe564bb: Download complete
[2021-04-15T13:20:29.481Z] 595b0fe564bb: Pull complete
[2021-04-15T13:20:29.481Z] Digest: sha256:8f18fae117ec6e5777cc62ba78cbb3be10a8a38639ccfb949521abd95c8301a4
[2021-04-15T13:20:29.481Z] Status: Downloaded newer image for arm64v8/alpine:3
[2021-04-15T13:20:32.076Z] Change permissions with write access of all files inside the specific folder
[2021-04-15T13:20:32.799Z] Running in /var/lib/jenkins/workspace/PR-24790-12-d24086b0-0153-4956-a02a-26a4c14dfcd6/src/github.com/elastic/beats/build
[2021-04-15T13:20:33.671Z] + rm -rf ve
[2021-04-15T13:20:33.671Z] + find . -type d -name vendor -exec rm -r {} ;
[2021-04-15T13:20:34.756Z] + python .ci/scripts/pre_archive_test.py
[2021-04-15T13:20:34.756Z] Copy ./libbeat/build into build/libbeat/build
[2021-04-15T13:20:34.960Z] Running in /var/lib/jenkins/workspace/PR-24790-12-d24086b0-0153-4956-a02a-26a4c14dfcd6/src/github.com/elastic/beats/build
[2021-04-15T13:20:35.039Z] Recording test results
[2021-04-15T13:20:41.625Z] [Checks API] No suitable checks publisher found.
[2021-04-15T13:20:42.589Z] + tar --version
[2021-04-15T13:20:43.733Z] + tar --exclude=test-build-artifacts-libbeat-arm-arm-tgz -czf test-build-artifacts-libbeat-arm-arm-tgz .
[2021-04-15T13:20:44.960Z] + gsutil --version
[2021-04-15T13:20:44.960Z] /var/lib/jenkins/workspace/PR-24790-12-d24086b0-0153-4956-a02a-26a4c14dfcd6/src/github.com/elastic/beats/build@tmp/durable-96b17f42/script.sh: 1: /var/lib/jenkins/workspace/PR-24790-12-d24086b0-0153-4956-a02a-26a4c14dfcd6/src/github.com/elastic/beats/build@tmp/durable-96b17f42/script.sh: gsutil: not found
[2021-04-15T13:20:45.214Z] Running in /var/lib/jenkins/workspace/PR-24790-12-d24086b0-0153-4956-a02a-26a4c14dfcd6/src/github.com/elastic/beats/build@tmp
[2021-04-15T13:20:46.109Z] + wget --version
[2021-04-15T13:20:47.414Z] + wget -q -O gsutil.tar.gz https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-319.0.0-linux-x86_64.tar.gz
[2021-04-15T13:20:49.022Z] + tar -xpf gsutil.tar.gz --strip-components=1
[2021-04-15T13:20:52.794Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-04-15T13:20:54.039Z] + gcloud auth activate-service-account --key-file ****
[2021-04-15T13:20:55.750Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-04-15T13:20:57.067Z] + gsutil -m -q cp -a public-read test-build-artifacts-libbeat-arm-arm-tgz gs://beats-ci-temp/Beats/beats/PR-24790-12
[2021-04-15T13:21:01.703Z] + python .ci/scripts/search_system_tests.py
[2021-04-15T13:21:01.990Z] [INFO] system-tests=''. If no empty then let's create a tarball
[2021-04-15T13:21:02.943Z] + go clean -modcache
[2021-04-15T13:21:06.519Z] Cleaning up /var/lib/jenkins/workspace/PR-24790-12-d24086b0-0153-4956-a02a-26a4c14dfcd6
[2021-04-15T13:21:06.519Z] Client: Docker Engine - Community
[2021-04-15T13:21:06.519Z]  Version:           20.10.3
[2021-04-15T13:21:06.519Z]  API version:       1.41
[2021-04-15T13:21:06.519Z]  Go version:        go1.13.15
[2021-04-15T13:21:06.519Z]  Git commit:        48d30b5
[2021-04-15T13:21:06.519Z]  Built:             Fri Jan 29 14:33:34 2021
[2021-04-15T13:21:06.519Z]  OS/Arch:           linux/arm64
[2021-04-15T13:21:06.519Z]  Context:           default
[2021-04-15T13:21:06.519Z]  Experimental:      true
[2021-04-15T13:21:06.519Z] 
[2021-04-15T13:21:06.519Z] Server: Docker Engine - Community
[2021-04-15T13:21:06.519Z]  Engine:
[2021-04-15T13:21:06.519Z]   Version:          20.10.3
[2021-04-15T13:21:06.519Z]   API version:      1.41 (minimum version 1.12)
[2021-04-15T13:21:06.519Z]   Go version:       go1.13.15
[2021-04-15T13:21:06.519Z]   Git commit:       46229ca
[2021-04-15T13:21:06.519Z]   Built:            Fri Jan 29 14:31:49 2021
[2021-04-15T13:21:06.519Z]   OS/Arch:          linux/arm64
[2021-04-15T13:21:06.519Z]   Experimental:     false
[2021-04-15T13:21:06.519Z]  containerd:
[2021-04-15T13:21:06.519Z]   Version:          1.4.4
[2021-04-15T13:21:06.519Z]   GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
[2021-04-15T13:21:06.519Z]  runc:
[2021-04-15T13:21:06.519Z]   Version:          1.0.0-rc93
[2021-04-15T13:21:06.519Z]   GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
[2021-04-15T13:21:06.519Z]  docker-init:
[2021-04-15T13:21:06.519Z]   Version:          0.19.0
[2021-04-15T13:21:06.519Z]   GitCommit:        de40ad0
[2021-04-15T13:21:06.519Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-04-15T13:21:07.716Z] Change permissions with write access of all files inside the specific folder
[2021-04-15T13:21:08.480Z] Running in /var/lib/jenkins/workspace/PR-24790-12-d24086b0-0153-4956-a02a-26a4c14dfcd6
[2021-04-15T13:21:14.521Z] Failed in branch libbeat-arm-arm
[2021-04-15T13:21:14.581Z] Stage "Extended" skipped due to earlier failure(s)
[2021-04-15T13:21:14.613Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-04-15T13:21:14.664Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-24790/src/github.com/elastic/beats
[2021-04-15T13:21:14.969Z] Running on worker-1244230 in /var/lib/jenkins/workspace/Beats_beats_PR-24790
[2021-04-15T13:21:15.007Z] [INFO] getVaultSecret: Getting secrets
[2021-04-15T13:21:15.105Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-04-15T13:21:17.105Z] + chmod 755 generate-build-data.sh
[2021-04-15T13:21:17.105Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24790/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24790/runs/12 FAILURE 1558041
[2021-04-15T13:21:17.105Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24790/runs/12/steps/?limit=10000 -o steps-info.json
[2021-04-15T13:21:18.762Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24790/runs/12/tests/?status=FAILED -o tests-errors.json
[2021-04-15T13:21:18.762Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24790/runs/12/log/ -o pipeline-log.txt

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test Results
Failed 1
Passed 2395
Skipped 3
Total 2399

Genuine test errors 1

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

  • Name: TestMonitorSample – github.com/elastic/beats/v7/libbeat/metric/system/cpu

@v1v
Copy link
Member Author

v1v commented Mar 26, 2021

/test ext auditbeat

@v1v
Copy link
Member Author

v1v commented Apr 6, 2021

/test

v1v and others added 2 commits April 7, 2021 11:48
…eline

* upstream/master:
  CI: curl seems to be available but where cannot find it (elastic#24965)
  [mergify] backport automation to 7.x, 7.12 and 7.11 (elastic#24608)
  Only show deprecation warnings for CN-based verification once (elastic#24948)
  CI: use ubuntu-20 (elastic#24963)
  Debug empty root field instead of error (elastic#24966)
  Kubernetes_secrets provider improvements (elastic#24912)
@v1v v1v added the skip-ci Skip the build in the CI but linting label Apr 7, 2021
@v1v
Copy link
Member Author

v1v commented Apr 7, 2021

/test

@v1v v1v added Team:Automation Label for the Observability productivity team and removed needs_team Indicates that the issue/PR needs a Team:* label skip-ci Skip the build in the CI but linting labels Apr 7, 2021
@v1v v1v self-assigned this Apr 7, 2021
@v1v
Copy link
Member Author

v1v commented Apr 10, 2021

/test

2 similar comments
@v1v
Copy link
Member Author

v1v commented Apr 13, 2021

/test

@v1v
Copy link
Member Author

v1v commented Apr 15, 2021

/test

@v1v
Copy link
Member Author

v1v commented Apr 15, 2021

I'm tempted to close this implementation which was quite intrusive compare to the new one

@v1v v1v closed this Apr 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Team:Automation Label for the Observability productivity team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants