Filter binaries by platform when EXTERNAL=true #1908
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Not all binaries are available for every platform, and if we try to download a binary from an artifact URL that doesn't exist the build fails.
cmacknz
commented
Dec 7, 2022
7 tasks
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
💚 Flaky test reportTests succeeded. 🤖 GitHub commentsExpand to view the GitHub comments
To re-run your PR in the CI, just comment with:
|
|
/test |
🌐 Coverage report
|
oren-zohar
requested changes
Dec 8, 2022
There was a problem hiding this comment.
Thanks for this fix @cmacknz. I think the issue here is rooted in the error handing in e2e-testing's FetchBeatsBinary, it shouldn't panic, and instead, it should propagate the error for outside handling. And on the agent side, whenever there's an error with fetching a specific artifact, I would suggest just printing it and moving forward with the build.
fortunately, I think this issue is solved with the latest e2e-testing package, this is what I did to verify:
- Update package dependency to
github.com/elastic/e2e-testing v1.99.2-0.20221205111528-ade3c840d0c0 - Uncommented
cloudbeatfrom the list of supported beats (I'll do it on my PR) - changed the panic here to just printing the error:
https://github.com/elastic/elastic-agent/blob/0e05b706399cfa380874ed1774315feb8440c475/magefile.go#LL668-L671C7
and these are the logs for running build command that is unsupported by both cloudbeat and APM
PLATFORMS=darwin/arm64 PACKAGES="docker" EXTERNAL=true SNAPSHOT=true mage -v packageLogs:
> PLATFORMS=darwin/arm64 PACKAGES="docker" EXTERNAL=true SNAPSHOT=true mage -v package
Running target: Package
WARN[0001] Download could not be found at the Elastic downloads API elapsedTime=384.747708ms error="HEAD request failed with 404" resolver="{ false map[] https://artifacts.elastic.co/downloads/beats/auditbeat/auditbeat-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz}" retry=1 statusEndpoint="https://artifacts.elastic.co/downloads/beats/auditbeat/auditbeat-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz"
WARN[0001] Object not found. Trying with another download resolver resolver="&{beats auditbeat-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz auditbeat}"
downloaded binaries on location: /Users/orenzohar/GolandProjects/oren-zohar/forks/elastic-agent/build/distributions/elastic-agent-drop/archives/darwin-aarch64.tar.gz/auditbeat-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz.sha512
WARN[0014] Download could not be found at the Elastic downloads API elapsedTime=211.779875ms error="HEAD request failed with 404" resolver="{ false map[] https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz}" retry=1 statusEndpoint="https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz"
WARN[0014] Object not found. Trying with another download resolver resolver="&{beats filebeat-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz filebeat}"
downloaded binaries on location: /Users/orenzohar/GolandProjects/oren-zohar/forks/elastic-agent/build/distributions/elastic-agent-drop/archives/darwin-aarch64.tar.gz/filebeat-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz.sha512
WARN[0027] Download could not be found at the Elastic downloads API elapsedTime=202.77375ms error="HEAD request failed with 404" resolver="{ false map[] https://artifacts.elastic.co/downloads/beats/heartbeat/heartbeat-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz}" retry=1 statusEndpoint="https://artifacts.elastic.co/downloads/beats/heartbeat/heartbeat-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz"
WARN[0027] Object not found. Trying with another download resolver resolver="&{beats heartbeat-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz heartbeat}"
downloaded binaries on location: /Users/orenzohar/GolandProjects/oren-zohar/forks/elastic-agent/build/distributions/elastic-agent-drop/archives/darwin-aarch64.tar.gz/heartbeat-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz.sha512
WARN[0037] Download could not be found at the Elastic downloads API elapsedTime=212.845042ms error="HEAD request failed with 404" resolver="{ false map[] https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz}" retry=1 statusEndpoint="https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz"
WARN[0037] Object not found. Trying with another download resolver resolver="&{beats metricbeat-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz metricbeat}"
downloaded binaries on location: /Users/orenzohar/GolandProjects/oren-zohar/forks/elastic-agent/build/distributions/elastic-agent-drop/archives/darwin-aarch64.tar.gz/metricbeat-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz.sha512
WARN[0052] Download could not be found at the Elastic downloads API elapsedTime=217.831958ms error="HEAD request failed with 404" resolver="{ false map[] https://artifacts.elastic.co/downloads/beats/osquerybeat/osquerybeat-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz}" retry=1 statusEndpoint="https://artifacts.elastic.co/downloads/beats/osquerybeat/osquerybeat-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz"
WARN[0052] Object not found. Trying with another download resolver resolver="&{beats osquerybeat-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz osquerybeat}"
downloaded binaries on location: /Users/orenzohar/GolandProjects/oren-zohar/forks/elastic-agent/build/distributions/elastic-agent-drop/archives/darwin-aarch64.tar.gz/osquerybeat-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz.sha512
WARN[0068] Download could not be found at the Elastic downloads API elapsedTime=196.64ms error="HEAD request failed with 404" resolver="{ false map[] https://artifacts.elastic.co/downloads/beats/packetbeat/packetbeat-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz}" retry=1 statusEndpoint="https://artifacts.elastic.co/downloads/beats/packetbeat/packetbeat-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz"
WARN[0068] Object not found. Trying with another download resolver resolver="&{beats packetbeat-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz packetbeat}"
downloaded binaries on location: /Users/orenzohar/GolandProjects/oren-zohar/forks/elastic-agent/build/distributions/elastic-agent-drop/archives/darwin-aarch64.tar.gz/packetbeat-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz.sha512
WARN[0079] Download could not be found at the Elastic downloads API elapsedTime=210.837166ms error="HEAD request failed with 404" resolver="{ false map[] https://artifacts.elastic.co/downloads/beats/apm-server/apm-server-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz}" retry=1 statusEndpoint="https://artifacts.elastic.co/downloads/beats/apm-server/apm-server-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz"
WARN[0079] Object not found. Trying with another download resolver resolver="&{beats apm-server-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz apm-server}"
ERRO[0079] object not found in Artifact API artifact=apm-server name=apm-server-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz version=8.7.0-SNAPSHOT
ERRO[0079] Object not found. There is no other download resolver
downloaded binaries on location:
WARN[0080] Download could not be found at the Elastic downloads API elapsedTime=207.659334ms error="HEAD request failed with 404" resolver="{ false map[] https://artifacts.elastic.co/downloads/beats/cloudbeat/cloudbeat-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz}" retry=1 statusEndpoint="https://artifacts.elastic.co/downloads/beats/cloudbeat/cloudbeat-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz"
WARN[0080] Object not found. Trying with another download resolver resolver="&{beats cloudbeat-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz cloudbeat}"
ERRO[0080] object not found in Artifact API artifact=cloudbeat name=cloudbeat-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz version=8.7.0-SNAPSHOT
ERRO[0080] Object not found. There is no other download resolver
downloaded binaries on location:
WARN[0080] Download could not be found at the Elastic downloads API elapsedTime=201.548834ms error="HEAD request failed with 404" resolver="{ false map[] https://artifacts.elastic.co/downloads/beats/elastic-agent-shipper/elastic-agent-shipper-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz}" retry=1 statusEndpoint="https://artifacts.elastic.co/downloads/beats/elastic-agent-shipper/elastic-agent-shipper-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz"
WARN[0080] Object not found. Trying with another download resolver resolver="&{beats elastic-agent-shipper-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz elastic-agent-shipper}"
downloaded binaries on location: /Users/orenzohar/GolandProjects/oren-zohar/forks/elastic-agent/build/distributions/elastic-agent-drop/archives/darwin-aarch64.tar.gz/elastic-agent-shipper-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz.sha512
WARN[0084] Download could not be found at the Elastic downloads API elapsedTime=203.390375ms error="HEAD request failed with 404" resolver="{ false map[] https://artifacts.elastic.co/downloads/beats/endpoint-security/endpoint-security-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz}" retry=1 statusEndpoint="https://artifacts.elastic.co/downloads/beats/endpoint-security/endpoint-security-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz"
WARN[0084] Object not found. Trying with another download resolver resolver="&{beats endpoint-security-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz endpoint-security}"
downloaded binaries on location: /Users/orenzohar/GolandProjects/oren-zohar/forks/elastic-agent/build/distributions/elastic-agent-drop/archives/darwin-aarch64.tar.gz/endpoint-security-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz.sha512
WARN[0095] Download could not be found at the Elastic downloads API elapsedTime=206.781167ms error="HEAD request failed with 404" resolver="{ false map[] https://artifacts.elastic.co/downloads/beats/fleet-server/fleet-server-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz}" retry=1 statusEndpoint="https://artifacts.elastic.co/downloads/beats/fleet-server/fleet-server-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz"
WARN[0095] Object not found. Trying with another download resolver resolver="&{beats fleet-server-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz fleet-server}"
downloaded binaries on location: /Users/orenzohar/GolandProjects/oren-zohar/forks/elastic-agent/build/distributions/elastic-agent-drop/archives/darwin-aarch64.tar.gz/fleet-server-8.7.0-SNAPSHOT-darwin-aarch64.tar.gz.sha512
exec: go "list" "-m"
Found Elastic Beats dir at /Users/orenzohar/GolandProjects/oren-zohar/forks/elastic-agent
elastic_beat_agent_binaries package spec loaded from [/Users/orenzohar/GolandProjects/oren-zohar/forks/elastic-agent/dev-tools/packaging/packages.yml]
Running dependency: Update
Running dependency: Config
Running dependency: configYML
>> Building elastic-agent.yml for linux/amd64
>> Building elastic-agent.reference.yml for linux/amd64
>> Building elastic-agent.docker.yml for linux/amd64
Running dependency: BuildPGP
>> BuildPGP from GPG-KEY-elasticsearch to internal/pkg/release/pgp.go
exec: go "run" "dev-tools/cmd/buildpgp/build_pgp.go" "--in" "GPG-KEY-elasticsearch" "--out" "internal/pkg/release/pgp.go"
Running dependency: BuildFleetCfg
>> BuildFleetCfg _meta/elastic-agent.fleet.yml to internal/pkg/agent/application/configuration_embed.go
exec: go "run" "dev-tools/cmd/buildfleetcfg/buildfleetcfg.go" "--in" "_meta/elastic-agent.fleet.yml" "--out" "internal/pkg/agent/application/configuration_embed.go"
Running dependency: CrossBuildGoDaemon
Skipping cross-build of target=buildGoDaemon because platforms list is empty.
Running dependency: CrossBuild
exec: docker "info" "-f" "{{ json .}}"
Running dependency: github.com/elastic/elastic-agent/dev-tools/mage.CrossBuild.func1
exec: go "mod" "download"
Running dependency: github.com/elastic/elastic-agent/dev-tools/mage.buildMage
exec: mage "-f" "-goos=linux" "-goarch=arm64" "-compile" "build/mage-linux-arm64"
...Rather than panicing when we attempt to download an artifact on an unsupported platform, we ignore the error and log that we are skipping the download.
|
Thanks @oren-zohar, didn't think to check that this was fixed in the e2e-testing package. That simplifies things nicely, although there are still some changes to the e2e-testing package we could follow up with. Setting the log level could be easier, and a defined error type for NotFound would be nice. |
cmacknz
commented
Dec 8, 2022
AndersonQ
reviewed
Dec 12, 2022
AndersonQ
requested changes
Dec 12, 2022
There was a problem hiding this comment.
#1908 (comment) isn't exacly a blocker, but I'd like to see what you think @cmacknz
Co-authored-by: Anderson Queiroz <me@andersonq.me>
|
SonarCloud Quality Gate failed.
|
AndersonQ
approved these changes
Dec 13, 2022
|
@oren-zohar after the latest update are there any changes you'd still like made to this PR? |
|
@cmacknz let me recheck this and I'll update you |
|
Everything works as expected with cloudbeat, thanks @cmacknz How I tested it SNAPSHOT=true DEV=true EXTERNAL=true PLATFORMS=linux/amd64 TYPES=docker mage package -vthis platform is supported by cloudbeat and everything worked as expected results with > SNAPSHOT=true DEV=true EXTERNAL=true PLATFORMS=windows/amd64 TYPES=docker mage package -v
downloaded binaries on /Users/orenzohar/GolandProjects/oren-zohar/forks/elastic-agent/build/distributions/elastic-agent-drop/archives/windows-x86_64.zip/auditbeat-8.7.0-SNAPSHOT-windows-x86_64.zip.sha512
downloaded binaries on /Users/orenzohar/GolandProjects/oren-zohar/forks/elastic-agent/build/distributions/elastic-agent-drop/archives/windows-x86_64.zip/filebeat-8.7.0-SNAPSHOT-windows-x86_64.zip.sha512
downloaded binaries on /Users/orenzohar/GolandProjects/oren-zohar/forks/elastic-agent/build/distributions/elastic-agent-drop/archives/windows-x86_64.zip/heartbeat-8.7.0-SNAPSHOT-windows-x86_64.zip.sha512
downloaded binaries on /Users/orenzohar/GolandProjects/oren-zohar/forks/elastic-agent/build/distributions/elastic-agent-drop/archives/windows-x86_64.zip/metricbeat-8.7.0-SNAPSHOT-windows-x86_64.zip.sha512
downloaded binaries on /Users/orenzohar/GolandProjects/oren-zohar/forks/elastic-agent/build/distributions/elastic-agent-drop/archives/windows-x86_64.zip/osquerybeat-8.7.0-SNAPSHOT-windows-x86_64.zip.sha512
downloaded binaries on /Users/orenzohar/GolandProjects/oren-zohar/forks/elastic-agent/build/distributions/elastic-agent-drop/archives/windows-x86_64.zip/packetbeat-8.7.0-SNAPSHOT-windows-x86_64.zip.sha512
Downloading cloudbeat: unsupported on windows/amd64, skipping
... |
oren-zohar
approved these changes
Dec 21, 2022
mergify bot
pushed a commit
that referenced
this pull request
Dec 21, 2022
* Filter binaries by platform when EXTERNAL=true Not all binaries are available for every platform, and if we try to download a binary from an artifact URL that doesn't exist the build fails. * Update e2e-testing/downloads, handle download errs. Rather than panicing when we attempt to download an artifact on an unsupported platform, we ignore the error and log that we are skipping the download. * Remove beats dependency. * Remove redundant word from log Co-authored-by: Anderson Queiroz <me@andersonq.me> Co-authored-by: Anderson Queiroz <me@andersonq.me> (cherry picked from commit a0e6108)
cmacknz
added a commit
that referenced
this pull request
Dec 22, 2022
* Filter binaries by platform when EXTERNAL=true Not all binaries are available for every platform, and if we try to download a binary from an artifact URL that doesn't exist the build fails. * Update e2e-testing/downloads, handle download errs. Rather than panicing when we attempt to download an artifact on an unsupported platform, we ignore the error and log that we are skipping the download. * Remove beats dependency. * Remove redundant word from log Co-authored-by: Anderson Queiroz <me@andersonq.me> Co-authored-by: Anderson Queiroz <me@andersonq.me> (cherry picked from commit a0e6108) Co-authored-by: Craig MacKenzie <craig.mackenzie@elastic.co>
pierrehilbert
added a commit
that referenced
this pull request
Aug 1, 2023
## What does this PR do? Adding cloudbeat to external flow, only supports `linux/amd64` and `linux/arm64` Following [this comment](#1716 (comment)) by @cmacknz ## Why is it important? Will improve dev flow for cloudbeat team ## Checklist - [x] My code follows the style guidelines of this project - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] I have made corresponding change to the default configuration files - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have added an entry in `CHANGELOG.next.asciidoc` or `CHANGELOG-developer.next.asciidoc`. ## Author's Checklist - [ ] tested with #1908 ## Related issues - Requires #1908 --------- Co-authored-by: Pierre HILBERT <pierre.hilbert@elastic.co>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Not all binaries are available for every platform, and if we try to download a binary from an artifact URL that doesn't exist the build fails.
To reproduce the problem this solves, run
EXTERNAL=true SNAPSHOT=true PLATFORMS=darwin/arm64 PACKAGES=tar.gz mage -v packageon main. This is what I use to package the agent for my M1 Mac regularly and APM server not being published for arm64 causes the build to fail when fetching from the artifacts API.This makes it so I don't have to manually remove APM server from the package list build every time I want build the agent.