Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial nomad support #23324

Closed
wants to merge 89 commits into from
Closed

Conversation

jsoriano
Copy link
Member

@jsoriano jsoriano commented Dec 30, 2020

Continues with #14954.

Differences over original PR:

Co-Authored-By: Jorge Luis Betancourt Gonzalez jorge-luis.betancourt@trivago.com

jorgelbg and others added 30 commits August 1, 2019 17:09
Add a test for the basic hint features
Add tests for the emition of events
Add tests for the emition of events
…llocations

Add local constants to track the possible status of the allocations
On older nomad versions (0.8.4) the `NodeName` attribute of the allocation is empty. This means that sometimes we cannot assign a proper `host` to the event. As a workaround we use the `NodeID` to get the the name from the actual client node.
This is a workaround for Nomad v0.8 that doesn't provide the NodeName directly in the allocation object. We use the NodeID to fetch it from the API.
- WIP emit only one task metadata event.
- Rename the matchers/indexers of the add_nomad_metadata processor to match the Nomad lingo.
- Rename `meta.meta` to `meta.tasks` and fix the tests.
- Add the main import to the nomad provider in the cmd tool.
WIP patch for the unchanged allocations and avoids triggering new harvesters for those allocations that were previously discovered.
- Rename `uuid` field to `alloc_id`.
- Use WatchOptions.RefreshInterval (SyncInterval on the config) for the sync interval of the watcher
jorgelbg and others added 9 commits November 16, 2020 23:02
After PR elastic#20281 was merged, the signature of the `autodiscover.ProviderBuilder` changed to accept an additional name (string) parameter.
After catching up with the latest changes in master these files changed and CI was complaining about the difference.
* Rename host to node for filtering, following nomad terminology.
* Pass address, namespace, region and secret ids to clients.
@jsoriano jsoriano added the Team:Integrations Label for the Integrations team label Dec 30, 2020
@jsoriano jsoriano self-assigned this Dec 30, 2020
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Dec 30, 2020
@botelastic
Copy link

botelastic bot commented Dec 30, 2020

This pull request doesn't have a Team:<team> label.

@elasticmachine
Copy link
Collaborator

elasticmachine commented Dec 30, 2020

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #23324 updated

  • Start Time: 2020-12-30T16:15:57.056+0000

  • Duration: 90 min 40 sec

Test stats 🧪

Test Results
Failed 0
Passed 17550
Skipped 1385
Total 18935

Steps errors 2

Expand to view the steps failures

Terraform Apply on x-pack/metricbeat/module/aws
  • Took 0 min 15 sec . View more details on here
  • Description: terraform apply -auto-approve
Terraform Apply on x-pack/metricbeat/module/aws
  • Took 0 min 15 sec . View more details on here
  • Description: terraform apply -auto-approve

Log output

Expand to view the last 100 lines of log output

[2020-12-30T17:07:17.868Z] + DOCKER_COMPOSE_VERSION=1.21.0
[2020-12-30T17:07:17.868Z] + HOME=/var/lib/jenkins/workspace/Beats_beats_PR-23324
[2020-12-30T17:07:17.868Z] + command -v docker-compose
[2020-12-30T17:07:17.868Z] /usr/local/bin/docker-compose
[2020-12-30T17:07:17.868Z] + echo 'Found docker-compose. Checking version..'
[2020-12-30T17:07:17.869Z] Found docker-compose. Checking version..
[2020-12-30T17:07:17.869Z] ++ docker-compose --version
[2020-12-30T17:07:17.869Z] ++ awk '{print $3}'
[2020-12-30T17:07:17.869Z] ++ sed s/,//
[2020-12-30T17:07:18.436Z] + FOUND_DOCKER_COMPOSE_VERSION=1.25.5
[2020-12-30T17:07:18.436Z] + '[' 1.25.5 == 1.21.0 ']'
[2020-12-30T17:07:18.436Z] + echo 'UNMET DEP: Installing docker-compose'
[2020-12-30T17:07:18.436Z] UNMET DEP: Installing docker-compose
[2020-12-30T17:07:18.436Z] + DC_CMD=/var/lib/jenkins/workspace/Beats_beats_PR-23324/bin/docker-compose
[2020-12-30T17:07:18.437Z] + mkdir -p /var/lib/jenkins/workspace/Beats_beats_PR-23324/bin
[2020-12-30T17:07:18.437Z] ++ uname -s
[2020-12-30T17:07:18.437Z] ++ uname -m
[2020-12-30T17:07:18.437Z] + curl -sSLo /var/lib/jenkins/workspace/Beats_beats_PR-23324/bin/docker-compose https://github.com/docker/compose/releases/download/1.21.0/docker-compose-Linux-x86_64
[2020-12-30T17:07:19.001Z] + chmod +x /var/lib/jenkins/workspace/Beats_beats_PR-23324/bin/docker-compose
[2020-12-30T17:07:19.001Z] + .ci/scripts/install-terraform.sh
[2020-12-30T17:07:19.001Z] + MSG='environment variable missing.'
[2020-12-30T17:07:19.001Z] + TERRAFORM_VERSION=0.12.24
[2020-12-30T17:07:19.001Z] + HOME=/var/lib/jenkins/workspace/Beats_beats_PR-23324
[2020-12-30T17:07:19.001Z] + TERRAFORM_CMD=/var/lib/jenkins/workspace/Beats_beats_PR-23324/bin/terraform
[2020-12-30T17:07:19.001Z] ++ uname -s
[2020-12-30T17:07:19.001Z] ++ tr '[:upper:]' '[:lower:]'
[2020-12-30T17:07:19.001Z] + OS=linux
[2020-12-30T17:07:19.001Z] + command -v terraform
[2020-12-30T17:07:19.001Z] + echo 'UNMET DEP: Installing Terraform'
[2020-12-30T17:07:19.001Z] UNMET DEP: Installing Terraform
[2020-12-30T17:07:19.001Z] + mkdir -p /var/lib/jenkins/workspace/Beats_beats_PR-23324/bin
[2020-12-30T17:07:19.001Z] + curl -sSLo - https://releases.hashicorp.com/terraform/0.12.24/terraform_0.12.24_linux_amd64.zip
[2020-12-30T17:07:19.259Z] ++ dirname /var/lib/jenkins/workspace/Beats_beats_PR-23324/bin/terraform
[2020-12-30T17:07:19.259Z] + unzip -o /var/lib/jenkins/workspace/Beats_beats_PR-23324/bin/terraform.zip -d /var/lib/jenkins/workspace/Beats_beats_PR-23324/bin
[2020-12-30T17:07:19.259Z] Archive:  /var/lib/jenkins/workspace/Beats_beats_PR-23324/bin/terraform.zip
[2020-12-30T17:07:19.823Z]   inflating: /var/lib/jenkins/workspace/Beats_beats_PR-23324/bin/terraform  
[2020-12-30T17:07:19.823Z] + rm /var/lib/jenkins/workspace/Beats_beats_PR-23324/bin/terraform.zip
[2020-12-30T17:07:19.823Z] + chmod +x /var/lib/jenkins/workspace/Beats_beats_PR-23324/bin/terraform
[2020-12-30T17:07:19.823Z] + make mage
[2020-12-30T17:07:19.823Z] Installing mage v1.10.0.
[2020-12-30T17:07:20.757Z] go: downloading github.com/magefile/mage v1.10.0
[2020-12-30T17:07:22.129Z] /var/lib/jenkins/workspace/Beats_beats_PR-23324/.magefile cleaned
[2020-12-30T17:07:22.427Z] + git config --get user.email
[2020-12-30T17:07:22.427Z] + [ -z  ]
[2020-12-30T17:07:22.427Z] + git config --global user.email beatsmachine@users.noreply.github.com
[2020-12-30T17:07:22.428Z] + git config --global user.name beatsmachine
[2020-12-30T17:07:22.713Z] + go mod download
[2020-12-30T17:07:38.011Z] + .ci/scripts/terraform-cleanup.sh x-pack/metricbeat
[2020-12-30T17:07:38.011Z] + DIRECTORY=x-pack/metricbeat
[2020-12-30T17:07:38.011Z] + FAILED=0
[2020-12-30T17:07:38.011Z] ++ find x-pack/metricbeat -name terraform.tfstate
[2020-12-30T17:07:38.011Z] + for tfstate in $(find $DIRECTORY -name terraform.tfstate)
[2020-12-30T17:07:38.011Z] ++ dirname x-pack/metricbeat/module/aws/terraform.tfstate
[2020-12-30T17:07:38.011Z] + cd x-pack/metricbeat/module/aws
[2020-12-30T17:07:38.011Z] + terraform destroy -auto-approve
[2020-12-30T17:07:42.190Z] 
[2020-12-30T17:07:42.190Z] Destroy complete! Resources: 0 destroyed.
[2020-12-30T17:07:42.190Z] + cd -
[2020-12-30T17:07:42.190Z] /var/lib/jenkins/workspace/Beats_beats_PR-23324/src/github.com/elastic/beats/src/github.com/elastic/beats
[2020-12-30T17:07:42.190Z] + exit 0
[2020-12-30T17:07:42.508Z] Client: Docker Engine - Community
[2020-12-30T17:07:42.508Z]  Version:           20.10.1
[2020-12-30T17:07:42.508Z]  API version:       1.41
[2020-12-30T17:07:42.508Z]  Go version:        go1.13.15
[2020-12-30T17:07:42.508Z]  Git commit:        831ebea
[2020-12-30T17:07:42.508Z]  Built:             Tue Dec 15 04:34:59 2020
[2020-12-30T17:07:42.508Z]  OS/Arch:           linux/amd64
[2020-12-30T17:07:42.508Z]  Context:           default
[2020-12-30T17:07:42.508Z]  Experimental:      true
[2020-12-30T17:07:42.508Z] 
[2020-12-30T17:07:42.508Z] Server: Docker Engine - Community
[2020-12-30T17:07:42.508Z]  Engine:
[2020-12-30T17:07:42.508Z]   Version:          20.10.1
[2020-12-30T17:07:42.508Z]   API version:      1.41 (minimum version 1.12)
[2020-12-30T17:07:42.508Z]   Go version:       go1.13.15
[2020-12-30T17:07:42.508Z]   Git commit:       f001486
[2020-12-30T17:07:42.508Z]   Built:            Tue Dec 15 04:32:40 2020
[2020-12-30T17:07:42.508Z]   OS/Arch:          linux/amd64
[2020-12-30T17:07:42.508Z]   Experimental:     false
[2020-12-30T17:07:42.508Z]  containerd:
[2020-12-30T17:07:42.508Z]   Version:          1.4.3
[2020-12-30T17:07:42.508Z]   GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
[2020-12-30T17:07:42.508Z]  runc:
[2020-12-30T17:07:42.508Z]   Version:          1.0.0-rc92
[2020-12-30T17:07:42.508Z]   GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
[2020-12-30T17:07:42.508Z]  docker-init:
[2020-12-30T17:07:42.508Z]   Version:          0.19.0
[2020-12-30T17:07:42.508Z]   GitCommit:        de40ad0
[2020-12-30T17:08:12.222Z] Scheduling project: Beats » Beats Packaging » PR-23324
[2020-12-30T17:08:17.564Z] Starting building: Beats » Beats Packaging » PR-23324 #10
[2020-12-30T17:45:35.064Z] [INFO] For detailed information see: https://beats-ci.elastic.co/job/Beats/job/packaging/job/PR-23324/10/display/redirect
[2020-12-30T17:45:36.056Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-23324/src/github.com/elastic/beats
[2020-12-30T17:45:36.640Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-23324
[2020-12-30T17:45:36.677Z] [INFO] getVaultSecret: Getting secrets
[2020-12-30T17:45:36.733Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-12-30T17:45:37.334Z] + chmod 755 generate-build-data.sh
[2020-12-30T17:45:37.335Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-23324/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-23324/runs/9 FAILURE 5380020
[2020-12-30T17:45:37.885Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-23324/runs/9/steps/?limit=10000 -o steps-info.json
[2020-12-30T17:45:39.746Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-23324/runs/9/tests/?status=FAILED -o tests-errors.json
[2020-12-30T17:45:40.657Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-23324/runs/9/log/ -o pipeline-log.txt

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 17550
Skipped 1385
Total 18935

@jsoriano jsoriano mentioned this pull request Dec 30, 2020
4 tasks
@jsoriano jsoriano changed the title Autodiscover nomad Initial nomad support Dec 30, 2020
@jsoriano
Copy link
Member Author

jsoriano commented Jan 5, 2021

Changes cherry-picked in original PR, closing this one.

@jsoriano jsoriano closed this Jan 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Team:Integrations Label for the Integrations team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants