-
Notifications
You must be signed in to change notification settings - Fork 42
feat: support passing KIBANA_VERSION #905
feat: support passing KIBANA_VERSION #905
Conversation
💔 Tests Failed
Expand to view the summary
Build stats
Test stats 🧪
Trends 🧪Test errorsExpand to view the tests failures
|
Test | Results |
---|---|
Failed | 1 |
Passed | 140 |
Skipped | 0 |
Total | 141 |
Genuine test errors
💔 There are test failures but not known flaky tests, most likely a genuine test failure.
- Name:
Initializing / End-To-End Tests / ubuntu-18.04_fleet_agent_endpoint_integration / Deploying an Endpoint makes policies to appear in the Security App – Agent Endpoint Integration
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With a brief review, it LGTM;
Hi also, with regards to: I wanted to acknowledge it would be helpful to have an explicit, and easily edited list of the e2e 'suites' that are included when we make the test run. Is that implied or otherwise easily done? I didn't see (maybe reading too fast) where in the current changes that we could control the list. For instance, do we want it to run the separate Helm chart or other Beats tests? Maybe. Maybe not, it would be good to make it explicit. |
This PR is adding the support, not changing the existing behavior, as KIBANA_VERSION would be the same as the stack. The second part will be a jenkins pipeline listening for kibana events on PRs, as the comment I suggested. This is not yet in place. About the different tests suites, the new pipeline will/should trigger the fleet tests, only. Exactly the same as the nightly build does. |
It uses the generic-webhook-trigger to get Kibana PRs payloads, and with that information trigger the e2e tests build.
printPostContent: false, | ||
silentResponse: true, | ||
regexpFilterText: '$GT_REPO$GT_BODY', | ||
regexpFilterExpression: '^elastic/kibana/run-fleet-e2e-tests$' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The command to run the tests in Kibana would be /run-fleet-e2e-tests
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@EricDavisX this pipeline represents the way to interact with Kibana PRs. Will document it, but:
- In a kibana PR, add a comment with
/run-fleet-e2e-tests
. Only for elasticians - Using the UI: trigger the pipeline using as the only parameter the GIT SHA of the kibana PR. (We still have to capture the version of kibana for that commit SHA)
.ci/e2eKibana.groovy
Outdated
) | ||
} | ||
parameters { | ||
string(name: 'kibana_sha', defaultValue: "", description: "Commit to be used to run the Fleet E2E Tests. (e.g 1234567890abcdef)") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Support running the tests from the UI
.ci/e2eKibana.groovy
Outdated
string(name: 'runTestsSuites', value: suite), | ||
string(name: 'GITHUB_CHECK_NAME', value: env.GITHUB_CHECK_E2E_TESTS_NAME), | ||
string(name: 'GITHUB_CHECK_REPO', value: env.REPO), | ||
string(name: 'GITHUB_CHECK_SHA1', value: getCommit()), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The commit to be used is passed here, using https://github.com/elastic/e2e-testing/pull/905/files#diff-ffe928572b6fdb91893a16cfc20f4b449acb6733b00a2051bf59b2e4c51872f8R72-R78
) | ||
} | ||
parameters { | ||
string(name: 'kibana_pr', defaultValue: "master", description: "PR ID to use to build the Docker image. (e.g 10000)") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We are simplifying the manual trigger: simply use the PR ID, without PR/
} | ||
|
||
// we are going to use the 'pr12345' tag | ||
return "pr${params.kibana_pr}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The Docker tag generated by the kibanaDockerBuild step builds and pushes the pr12345
tag
return "pr${params.kibana_pr}" | ||
} | ||
|
||
def hasCommentAuthorWritePermissions(prId, commentId){ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Potentially reuse elastic/apm-pipeline-library#1064
* master: Remove the agent config file parameters for stand alone (elastic#983) Uniquify the stand-alone step for checking agent status (elastic#993)
I'm gonna merge this one, so that we can start using the new pipeline and detecting possible errors the soonest |
* chore: support returning different fallback Docker namespaces * chore: add KIBANA_VERSION variable * docs: document new Kibana variable * chore(ci): add regular to pipeline for Kibana PRs It uses the generic-webhook-trigger to get Kibana PRs payloads, and with that information trigger the e2e tests build. * chore: only process kibana events * chore: add PR author to the cause string * chore: print PR sha in cause string * chore: update Generic Trigger to match existing example * chore: support manually running the tests for a PR * chore: do not notify on slack on build greens * fix: rename JJB as it's not a multibranch pipeline * feat: build kibana image based on the PR or the UI params * chore: simplify triggering the pipeline from the UI * chore: simplify even more * chore: use stackVersion if kibana version is not set * docs: update docs about KIBANA_VERSION * chore: rename variable * chore: rename job * fix: misleading param description * fix: set the proper Docker namespace for Kibana (cherry picked from commit 7abccaf) # Conflicts: # .ci/Jenkinsfile # cli/config/compose/profiles/fleet/docker-compose.yml # e2e/_suites/fleet/README.md # e2e/_suites/fleet/ingest_manager_test.go # e2e/_suites/metricbeat/README.md
* chore: support returning different fallback Docker namespaces * chore: add KIBANA_VERSION variable * docs: document new Kibana variable * chore(ci): add regular to pipeline for Kibana PRs It uses the generic-webhook-trigger to get Kibana PRs payloads, and with that information trigger the e2e tests build. * chore: only process kibana events * chore: add PR author to the cause string * chore: print PR sha in cause string * chore: update Generic Trigger to match existing example * chore: support manually running the tests for a PR * chore: do not notify on slack on build greens * fix: rename JJB as it's not a multibranch pipeline * feat: build kibana image based on the PR or the UI params * chore: simplify triggering the pipeline from the UI * chore: simplify even more * chore: use stackVersion if kibana version is not set * docs: update docs about KIBANA_VERSION * chore: rename variable * chore: rename job * fix: misleading param description * fix: set the proper Docker namespace for Kibana (cherry picked from commit 7abccaf) # Conflicts: # .ci/Jenkinsfile # cli/config/compose/profiles/fleet/docker-compose.yml # e2e/_suites/fleet/README.md # e2e/_suites/fleet/ingest_manager_test.go # e2e/_suites/metricbeat/README.md
* chore: support returning different fallback Docker namespaces * chore: add KIBANA_VERSION variable * docs: document new Kibana variable * chore(ci): add regular to pipeline for Kibana PRs It uses the generic-webhook-trigger to get Kibana PRs payloads, and with that information trigger the e2e tests build. * chore: only process kibana events * chore: add PR author to the cause string * chore: print PR sha in cause string * chore: update Generic Trigger to match existing example * chore: support manually running the tests for a PR * chore: do not notify on slack on build greens * fix: rename JJB as it's not a multibranch pipeline * feat: build kibana image based on the PR or the UI params * chore: simplify triggering the pipeline from the UI * chore: simplify even more * chore: use stackVersion if kibana version is not set * docs: update docs about KIBANA_VERSION * chore: rename variable * chore: rename job * fix: misleading param description * fix: set the proper Docker namespace for Kibana (cherry picked from commit 7abccaf) # Conflicts: # .ci/Jenkinsfile # cli/config/compose/profiles/fleet/docker-compose.yml # e2e/_suites/fleet/README.md # e2e/_suites/fleet/ingest_manager_test.go # e2e/_suites/metricbeat/README.md
* feat: support passing KIBANA_VERSION (#905) * chore: support returning different fallback Docker namespaces * chore: add KIBANA_VERSION variable * docs: document new Kibana variable * chore(ci): add regular to pipeline for Kibana PRs It uses the generic-webhook-trigger to get Kibana PRs payloads, and with that information trigger the e2e tests build. * chore: only process kibana events * chore: add PR author to the cause string * chore: print PR sha in cause string * chore: update Generic Trigger to match existing example * chore: support manually running the tests for a PR * chore: do not notify on slack on build greens * fix: rename JJB as it's not a multibranch pipeline * feat: build kibana image based on the PR or the UI params * chore: simplify triggering the pipeline from the UI * chore: simplify even more * chore: use stackVersion if kibana version is not set * docs: update docs about KIBANA_VERSION * chore: rename variable * chore: rename job * fix: misleading param description * fix: set the proper Docker namespace for Kibana (cherry picked from commit 7abccaf) # Conflicts: # .ci/Jenkinsfile # cli/config/compose/profiles/fleet/docker-compose.yml # e2e/_suites/fleet/README.md # e2e/_suites/fleet/ingest_manager_test.go # e2e/_suites/metricbeat/README.md * fix: resolve conflicts * fix: remove unused variable * Revert "fix: remove unused variable" This reverts commit e20645a. * fix: do not remove kibana config yet Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com>
* feat: support passing KIBANA_VERSION (#905) * chore: support returning different fallback Docker namespaces * chore: add KIBANA_VERSION variable * docs: document new Kibana variable * chore(ci): add regular to pipeline for Kibana PRs It uses the generic-webhook-trigger to get Kibana PRs payloads, and with that information trigger the e2e tests build. * chore: only process kibana events * chore: add PR author to the cause string * chore: print PR sha in cause string * chore: update Generic Trigger to match existing example * chore: support manually running the tests for a PR * chore: do not notify on slack on build greens * fix: rename JJB as it's not a multibranch pipeline * feat: build kibana image based on the PR or the UI params * chore: simplify triggering the pipeline from the UI * chore: simplify even more * chore: use stackVersion if kibana version is not set * docs: update docs about KIBANA_VERSION * chore: rename variable * chore: rename job * fix: misleading param description * fix: set the proper Docker namespace for Kibana (cherry picked from commit 7abccaf) # Conflicts: # .ci/Jenkinsfile # cli/config/compose/profiles/fleet/docker-compose.yml # e2e/_suites/fleet/README.md # e2e/_suites/fleet/ingest_manager_test.go # e2e/_suites/metricbeat/README.md * fix: resolve conflicts * fix: remove unused variable * Revert "fix: remove unused variable" This reverts commit b192f74. * fix: do not remove kibana path yet Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com>
* feat: support passing KIBANA_VERSION (#905) * chore: support returning different fallback Docker namespaces * chore: add KIBANA_VERSION variable * docs: document new Kibana variable * chore(ci): add regular to pipeline for Kibana PRs It uses the generic-webhook-trigger to get Kibana PRs payloads, and with that information trigger the e2e tests build. * chore: only process kibana events * chore: add PR author to the cause string * chore: print PR sha in cause string * chore: update Generic Trigger to match existing example * chore: support manually running the tests for a PR * chore: do not notify on slack on build greens * fix: rename JJB as it's not a multibranch pipeline * feat: build kibana image based on the PR or the UI params * chore: simplify triggering the pipeline from the UI * chore: simplify even more * chore: use stackVersion if kibana version is not set * docs: update docs about KIBANA_VERSION * chore: rename variable * chore: rename job * fix: misleading param description * fix: set the proper Docker namespace for Kibana (cherry picked from commit 7abccaf) # Conflicts: # .ci/Jenkinsfile # cli/config/compose/profiles/fleet/docker-compose.yml # e2e/_suites/fleet/README.md # e2e/_suites/fleet/ingest_manager_test.go # e2e/_suites/metricbeat/README.md * fix: resolve conflicts * fix: remove unused variable * Revert "fix: remove unused variable" This reverts commit e69fe76. * fix: do not remove kibana config yet Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com>
* master: chore: add debug info for the payload (elastic#1044) chore: add debug traces for the webhook payload (elastic#1043) fix: wrong interpolation (elastic#1042) Update Elastic Agent to not use Kibana (elastic#1036) fix: apply X version for non-master branches (elastic#1037) fix: add NodeJS to PATH (elastic#1035) fix: use an agent when building kibana (elastic#1030) fix(jjb): use a branch that exists (elastic#1029) remove uninstall step (elastic#1017) fix: delay checking stale agent version until it's used (elastic#1016) fix: use same JJB than in custom kibana (elastic#1010) chore: simplify PR template (elastic#1011) feat: support passing KIBANA_VERSION (elastic#905) [mergify] assign the original author (elastic#1009) Remove the agent config file parameters for stand alone (elastic#983) Uniquify the stand-alone step for checking agent status (elastic#993)
* master: fix: check if kibana version is a commit (elastic#1047) fix: incorrect variable name (elastic#1046) fix: typo in parameters (elastic#1045) chore: add debug info for the payload (elastic#1044) chore: add debug traces for the webhook payload (elastic#1043) fix: wrong interpolation (elastic#1042) Update Elastic Agent to not use Kibana (elastic#1036) fix: apply X version for non-master branches (elastic#1037) fix: add NodeJS to PATH (elastic#1035) fix: use an agent when building kibana (elastic#1030) fix(jjb): use a branch that exists (elastic#1029) remove uninstall step (elastic#1017) fix: delay checking stale agent version until it's used (elastic#1016) fix: use same JJB than in custom kibana (elastic#1010) chore: simplify PR template (elastic#1011) feat: support passing KIBANA_VERSION (elastic#905) [mergify] assign the original author (elastic#1009)
What does this PR do?
This PR adds a way to instrument the test framework to consume a different Docker image for Kibana: instead of using the
STACK_VERSION
variable, Kibana container version will be managed byKIBANA_VERSION
. If it's not set, then the Docker image related to a Kibana PR will be used. This image is stored under theobservability-ci
Docker namespace.We are also adding a pipeline that is listening for a webhook on the Kibana repo, and when the comment
/run-fleet-e2e-tests
is added to a PR by an Elastician, then the new pipeline will be triggered building and pushing the Docker image for that PR, and finally using that image in the e2e tests. This new pipeline will allow running it manually, passing the PR ID as an input parameter.Why is it important?
It will support the ability of running the tests against Kibana PRs (specially for Fleet), using the Dockerised version of Kibana for the PR.
Checklist
make notice
in the proper directory)Author's Checklist
How to test this PR locally
Running a PR version of Kibana, i.e.
pr94204
, that we already know it was pushed to the registry:Running default version of Kibana, i.e.
8.0.0-SNAPSHOT
:Related issues
Follow-ups
We'd need to add a Jenkins pipeline listening for Kibana PRs, with the proposed synchronous workflow (@EricDavisX @ph, please confirm):
a. the author comments with
/run-fleet-e2e-tests
, orb. the modified paths affect Fleet code
Deploy my Kibana
, without deploying it to the test clusterKIBANA_VERSION=prXYZ