PoC: use pipeline generator #38162

v1v · 2024-03-04T08:52:24Z

Proposed commit message

WHAT: pipeline migration from Jenkins to Buildkite using a pipeline generator based on Python.
WHY: takes the concepts from [CI] Pipeline 2.0 for monorepos #20104 but for Buildkite

Tasks

Enable when conditions.
- GitHub comment - for individual stages and groups
- GitHub labels - for individual stages
- Changeset
- Branches
Use catalog to generate the pipeline itself, rather than calling a python script on a pipeline.yaml file.
Support draft PRs.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

mergify · 2024-03-04T08:53:17Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @v1v? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

elasticmachine · 2024-03-04T09:05:49Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Duration: 79 min 25 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

v1v · 2024-03-04T10:38:39Z

.buildkite/buildkite.yml

@@ -0,0 +1,44 @@
+projects:


What beats are onboarded to this automated pipeline generator.

These are folders from the top of the project. They should also host a similar buildkite.yml file.

auditbeat/buildkite.yml

v1v · 2024-03-04T10:43:17Z

auditbeat/buildkite.yml

+    #       - GitHub label <sub-stage-name> . i.e: integTest or unitTest-arm or unitTest-macos ...
+    extended:
+        # NOTE: stage name should be unique!
+        integTest:


This step can be forced to run using:

The GitHub comment /test auditbeat integTest

The GitHub label auditbeat-integTest

v1v · 2024-03-04T10:43:50Z

auditbeat/buildkite.yml

+        integTest:
+            command: "mage build integTest"
+            platform: "core-ubuntu-2004-aarch64"
+        integTest-arm:


This step can be forced to run using:

The GitHub comment /test auditbeat integTest-arm

The GitHub label auditbeat-integTest-arm

v1v · 2024-03-04T10:44:35Z

auditbeat/buildkite.yml

+    #   - on PRs if:
+    #       - GitHub comment /test auditbeat <sub-stage-name> . i.e: /test auditbeat integTest
+    #       - GitHub label <sub-stage-name> . i.e: integTest or unitTest-arm or unitTest-macos ...
+    extended:


All the stages in the extended meta-stage will be triggered for PRs using:

The GitHub comment /test auditbeat-extended

The GitHub label extended-auditbeat

pazone · 2024-03-04T12:29:00Z

If I understand correctly it allows us to keep the previously created Jenkinsfile.yml-based approach without any significant structural changes.
Let's figure out what else we need to do to make it work e.g. secrets management, hooks, conditional tools installation, etc.

v1v · 2024-03-04T15:38:03Z

buildkite test auditbeat

alexsapran · 2024-03-07T13:20:51Z

.buildkite/pipeline.py

+
+    def create_entity(self):
+        data = """
+{% if group.steps|length > 0 %}


Can we switch from templating the YAML to just doing something along the lines of

import yaml # ... yaml.dumps(self)

Sure, it was just a quick PoC to illustrate how this could work. Moving from one implementation to another is totally fine and makes sense if that's the way to move this forward :)

.buildkite/scripts/generate_pipeline.sh

…red tools

… work

pazone · 2024-03-15T00:41:41Z

Added:

changeset functionality,
aws and orka support,
applied beats images for jobs.
The pipeline generation is tested on filebeat and auditbeat. It works fine but I'm facing issues with filebeat crosscompile. It's not related to the pipeline generation logic.

I think we can merge this PR.

Further improvements:

change pipeline generation for comments (now it doesn't work properly)
unit tests for the pipeline.py

elasticmachine · 2024-03-20T12:39:26Z

💚 Build Succeeded

Buildkite Build
Commit: ca2c8fc

History

💚 Build #2419 succeeded 48a7851
💚 Build #2405 succeeded 3a1cc1b
💚 Build #2355 succeeded c839a67
💚 Build #2350 succeeded 996d322
💚 Build #2329 succeeded 79d7a48
💚 Build #2323 succeeded b15fc29

cc @v1v

elasticmachine · 2024-03-20T12:43:30Z

💚 Build Succeeded

Buildkite Build
Commit: ca2c8fc

History

💚 Build #1147 succeeded 48a7851
💚 Build #1133 succeeded 3a1cc1b
💚 Build #1083 succeeded c839a67
💚 Build #1042 succeeded 9dc603a
💔 Build #1002 failed 99dffd7
💔 Build #984 failed 6bfdd74

cc @v1v

elasticmachine · 2024-03-20T12:46:15Z

💚 Build Succeeded

Buildkite Build
Commit: ca2c8fc

History

💚 Build #2716 succeeded 48a7851
💚 Build #2702 succeeded 3a1cc1b
💔 Build #2652 failed c839a67
💔 Build #2649 failed 1dcce41
💔 Build #2647 failed 996d322
💔 Build #2638 failed d49d8c0

cc @v1v

elasticmachine · 2024-03-20T12:48:40Z

💚 Build Succeeded

Buildkite Build
Commit: ca2c8fc

History

💚 Build #1582 succeeded 48a7851
💚 Build #1568 succeeded 3a1cc1b
💚 Build #1518 succeeded c839a67
💚 Build #1477 succeeded 9dc603a
💚 Build #1437 succeeded 99dffd7
💚 Build #1419 succeeded 6bfdd74

cc @v1v

elasticmachine · 2024-03-20T12:50:45Z

💚 Build Succeeded

Buildkite Build
Commit: ca2c8fc

History

💚 Build #2426 succeeded 48a7851
💚 Build #2412 succeeded 3a1cc1b
💚 Build #2362 succeeded c839a67
💚 Build #2321 succeeded 9dc603a
💚 Build #2281 succeeded 99dffd7
💔 Build #2263 failed 6bfdd74

cc @v1v

elasticmachine · 2024-03-20T12:54:22Z

💚 Build Succeeded

Buildkite Build
Commit: ca2c8fc

History

💚 Build #1144 succeeded 48a7851
💚 Build #1130 succeeded 3a1cc1b
💚 Build #1080 succeeded c839a67
💚 Build #1039 succeeded 9dc603a
💚 Build #999 succeeded 99dffd7
💚 Build #981 succeeded 6bfdd74

cc @v1v

elasticmachine · 2024-03-20T12:59:23Z

💚 Build Succeeded

Buildkite Build
Commit: ca2c8fc

History

💚 Build #1143 succeeded 3a1cc1b
💚 Build #1093 succeeded c839a67
💚 Build #1052 succeeded 9dc603a
💚 Build #1012 succeeded 99dffd7
💚 Build #994 succeeded 6bfdd74
💚 Build #931 succeeded 2835ba1

cc @v1v

elasticmachine · 2024-03-20T13:04:43Z

💚 Build Succeeded

Buildkite Build
Commit: ca2c8fc

History

💔 Build #3623 failed 3a1cc1b
💚 Build #3573 succeeded c839a67
💚 Build #3532 succeeded 9dc603a
💚 Build #3492 succeeded 99dffd7
💚 Build #3474 succeeded 6bfdd74

cc @v1v

elasticmachine · 2024-03-20T13:07:50Z

💚 Build Succeeded

Buildkite Build
Commit: ca2c8fc

History

💚 Build #4055 succeeded 3a1cc1b
💔 Build #4005 failed c839a67
💔 Build #4002 failed 1dcce41
💔 Build #4000 failed 996d322
💔 Build #3991 failed d49d8c0
💔 Build #3979 failed 79d7a48

cc @v1v

norrietaylor

LGTM

This commit introduces a main pipeline generator similar to the one used with Jenkins #20104 to help with the migration to Buildkite. Co-authored-by: Pavel Zorin <pavel.zorin@elastic.co> Co-authored-by: Alexandros Sapranidis <alexandros@elastic.co> (cherry picked from commit 4b7467b)

This commit introduces a main pipeline generator similar to the one used with Jenkins #20104 to help with the migration to Buildkite. Co-authored-by: Pavel Zorin <pavel.zorin@elastic.co> Co-authored-by: Alexandros Sapranidis <alexandros@elastic.co> (cherry picked from commit 4b7467b) # Conflicts: # .buildkite/hooks/pre-command

.buildkite/pipeline.py

This commit introduces a main pipeline generator similar to the one used with Jenkins #20104 to help with the migration to Buildkite. Co-authored-by: Pavel Zorin <pavel.zorin@elastic.co> Co-authored-by: Alexandros Sapranidis <alexandros@elastic.co> (cherry picked from commit 4b7467b) # Conflicts: # .buildkite/hooks/pre-command Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

This commit introduces a main pipeline generator similar to the one used with Jenkins #20104 to help with the migration to Buildkite. Co-authored-by: Pavel Zorin <pavel.zorin@elastic.co> Co-authored-by: Alexandros Sapranidis <alexandros@elastic.co> (cherry picked from commit 4b7467b) Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Mar 4, 2024

mergify bot assigned v1v Mar 4, 2024

alexsapran requested a review from dliappis March 4, 2024 09:59

v1v commented Mar 4, 2024

View reviewed changes

v1v mentioned this pull request Mar 4, 2024

buildkite: change github comments #38173

Closed

6 tasks

alexsapran reviewed Mar 7, 2024

View reviewed changes

.buildkite/scripts/generate_pipeline.sh Outdated Show resolved Hide resolved

v1v and others added 11 commits March 13, 2024 14:42

buildkite: python generator

55e3666

for testing purposes

172c2bb

use intermediate script to run the pipeline generation with the requi…

59de509

…red tools

debug traces

306b2ff

fix typo

05fa8c4

support for PRs, labels and GH commands

340f6d1

support category in the steps so github comments for mandatory stages…

4b20d19

… work

avoid empty groups

596890d

refactored

df55062

changeset

0f4df5d

Changesets

d006b37

pazone force-pushed the feature/buildkite branch from fefe766 to d006b37 Compare March 13, 2024 18:38

pazone added 5 commits March 14, 2024 23:15

fix global variable

e51daa0

Use beats images

b92ffd7

added tool versions and enabled pre-command for beats pipeline

3449137

Use beets VM images for filebeat

ab9eecb

AWS And Orka support

b5cdc4c

pazone marked this pull request as ready for review March 15, 2024 00:46

jlind23 approved these changes Mar 20, 2024

View reviewed changes

norrietaylor approved these changes Mar 20, 2024

View reviewed changes

alexsapran merged commit 4b7467b into main Mar 20, 2024
63 of 67 checks passed

alexsapran deleted the feature/buildkite branch March 20, 2024 14:27

alexsapran added the backport-v8.13.0 Automated backport with mergify label Mar 20, 2024

mergify bot mentioned this pull request Mar 20, 2024

[8.13](backport #38162) PoC: use pipeline generator #38478

Merged

13 tasks

pazone added backport-v8.12.0 Automated backport with mergify backport-7.17 Automated backport to the 7.17 branch with mergify labels Mar 20, 2024

mergify bot mentioned this pull request Mar 20, 2024

[7.17](backport #38162) PoC: use pipeline generator #38485

Merged

13 tasks

mergify bot mentioned this pull request Mar 20, 2024

[8.12](backport #38162) PoC: use pipeline generator #38486

Merged

13 tasks

alexsapran added the Team:Ingest-EngProd label Mar 21, 2024

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 21, 2024

alexsapran added the ci label Mar 21, 2024

v1v commented Mar 21, 2024

View reviewed changes

.buildkite/pipeline.py Show resolved Hide resolved

v1v commented Mar 21, 2024

View reviewed changes

.buildkite/pipeline.py Show resolved Hide resolved

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PoC: use pipeline generator #38162

PoC: use pipeline generator #38162

v1v commented Mar 4, 2024 •

edited by pazone

Loading

mergify bot commented Mar 4, 2024

elasticmachine commented Mar 4, 2024 •

edited

Loading

Build stats

v1v Mar 4, 2024

v1v Mar 4, 2024

v1v Mar 4, 2024

v1v Mar 4, 2024

pazone commented Mar 4, 2024 •

edited

Loading

v1v commented Mar 4, 2024

alexsapran Mar 7, 2024

v1v Mar 7, 2024

pazone commented Mar 15, 2024 •

edited

Loading

elasticmachine commented Mar 20, 2024

elasticmachine commented Mar 20, 2024

elasticmachine commented Mar 20, 2024

elasticmachine commented Mar 20, 2024

elasticmachine commented Mar 20, 2024

elasticmachine commented Mar 20, 2024

elasticmachine commented Mar 20, 2024

elasticmachine commented Mar 20, 2024

elasticmachine commented Mar 20, 2024

norrietaylor left a comment

PoC: use pipeline generator #38162

PoC: use pipeline generator #38162

Conversation

v1v commented Mar 4, 2024 • edited by pazone Loading

Proposed commit message

Tasks

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

mergify bot commented Mar 4, 2024

elasticmachine commented Mar 4, 2024 • edited Loading

💚 Build Succeeded

Build stats

❕ Flaky test report

🤖 GitHub comments

v1v Mar 4, 2024

Choose a reason for hiding this comment

v1v Mar 4, 2024

Choose a reason for hiding this comment

v1v Mar 4, 2024

Choose a reason for hiding this comment

v1v Mar 4, 2024

Choose a reason for hiding this comment

pazone commented Mar 4, 2024 • edited Loading

v1v commented Mar 4, 2024

alexsapran Mar 7, 2024

Choose a reason for hiding this comment

v1v Mar 7, 2024

Choose a reason for hiding this comment

pazone commented Mar 15, 2024 • edited Loading

elasticmachine commented Mar 20, 2024

💚 Build Succeeded

History

elasticmachine commented Mar 20, 2024

💚 Build Succeeded

History

elasticmachine commented Mar 20, 2024

💚 Build Succeeded

History

elasticmachine commented Mar 20, 2024

💚 Build Succeeded

History

elasticmachine commented Mar 20, 2024

💚 Build Succeeded

History

elasticmachine commented Mar 20, 2024

💚 Build Succeeded

History

elasticmachine commented Mar 20, 2024

💚 Build Succeeded

History

elasticmachine commented Mar 20, 2024

💚 Build Succeeded

History

elasticmachine commented Mar 20, 2024

💚 Build Succeeded

History

norrietaylor left a comment

Choose a reason for hiding this comment

v1v commented Mar 4, 2024 •

edited by pazone

Loading

elasticmachine commented Mar 4, 2024 •

edited

Loading

pazone commented Mar 4, 2024 •

edited

Loading

pazone commented Mar 15, 2024 •

edited

Loading