[Elastic-Agent] Elastic Agent doesn't read sourceURI parameter from the policy.

[criamico]

Elastic Agent doesn't read sourceURI parameter from the policy.

I discovered this bug while testing #756. While the policy reloading itself works fine, the agent doesn't seem to read the sourceURI passed through the policy under agent.download.source_uri.

• Version: 8.4.0-SNAPSHOT
• Operating System: amd64 DOCKER - build from [Elastic-Agent] Added source uri reloading #756

Test case 1

I tried to upgrade the agent and would expect the agent to pick up the source URI from the policy, however it doesn't read it.

• Created In Kibana a new source download source object with a mock source URI https://test-registry.co

• Created a new agent policy and assigned the download source object to it

• Did a single agent update with no source URI. I had to disable some internal version checks in fleet UI

  POST kbn:/api/fleet/agents/64be65a3-996a-4eea-99f8-d0577540d5a8/upgrade
  {
    "version": "8.4.0"
  }
  

The agent is trying to update from the default registry - the custom source URI doesn't appear at all:

{"log.level":"info","@timestamp":"2022-07-21T10:42:49.312Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-07-21T10:42:49Z - message: Application: [64be65a3-996a-4eea-99f8-d0577540d5a8]: State changed to UPDATING: Update to version '8.4.0' started - type: 'STATE' - sub_type: 'UPDATING'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-07-21T10:43:18.547Z","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2022-07-21T10:43:18Z - message: Application: [64be65a3-996a-4eea-99f8-d0577540d5a8]: State changed to FAILED: failed upgrade of agent binary: 3 errors occurred:\n\t* package '/usr/share/elastic-agent/state/data/downloads/elastic-agent-8.4.0-linux-x86_64.tar.gz' not found: open /usr/share/elastic-agent/state/data/downloads/elastic-agent-8.4.0-linux-x86_64.tar.gz: no such file or directory\n\t* call to 'https://snapshots.elastic.co/8.4.0-31269fd2/downloads/beats/elastic-agent/elastic-agent-8.4.0-linux-x86_64.tar.gz' returned unsuccessful status code: 404\n\t* call to 'https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.4.0-linux-x86_64.tar.gz' returned unsuccessful status code: 404\n\n - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-07-21T10:43:18.547Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":211},"message":"failed to dispatch actions, error: failed upgrade of agent binary: 3 errors occurred:\n\t* package '/usr/share/elastic-agent/state/data/downloads/elastic-agent-8.4.0-linux-x86_64.tar.gz' not found: open /usr/share/elastic-agent/state/data/downloads/elastic-agent-8.4.0-linux-x86_64.tar.gz: no such file or directory\n\t* call to 'https://snapshots.elastic.co/8.4.0-31269fd2/downloads/beats/elastic-agent/elastic-agent-8.4.0-linux-x86_64.tar.gz' returned unsuccessful status code: 404\n\t* call to 'https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.4.0-linux-x86_64.tar.gz' returned unsuccessful status code: 404\n\n","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-07-21T10:43:18.547Z","log.origin":{"file.name":"status/reporter.go","file.line":260},"message":"Elastic Agent status changed to \"error\": \"component gateway-617349ad: failed to dispatch actions, error: failed upgrade of agent binary: 3 errors occurred:\\n\\t* package '/usr/share/elastic-agent/state/data/downloads/elastic-agent-8.4.0-linux-x86_64.tar.gz' not found: open /usr/share/elastic-agent/state/data/downloads/elastic-agent-8.4.0-linux-x86_64.tar.gz: no such file or directory\\n\\t* call to 'https://snapshots.elastic.co/8.4.0-31269fd2/downloads/beats/elastic-agent/elastic-agent-8.4.0-linux-x86_64.tar.gz' returned unsuccessful status code: 404\\n\\t* call to 'https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.4.0-linux-x86_64.tar.gz' returned unsuccessful status code: 404\\n\\n\"","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-21T10:47:55.611Z","log.origin":{"file.name":"status/reporter.go","file.line":260},"message":"Elastic Agent status changed to \"online\": \"\"","ecs.version":"1.6.0"}

The expected behaviour would be trying to download the agent from https://test-registry.co, but it doesn't even appear in the logs.

Verified that in .fleet_actions there is no URI:

 {
        "_index": ".fleet-actions-7",
        "_id": "6eb3ac36-07de-4056-88f9-9979422d80fc",
        "_score": 1,
        "_source": {
          "@timestamp": "2022-07-21T14:16:36.388Z",
          "expiration": "2022-08-20T14:16:36.388Z",
          "agents": [
            "438e8fd3-7643-49b8-8a38-8bad0ee421f1"
          ],
          "action_id": "7e09cddc-e183-41fb-be9c-d99032262c44",
          "data": {
            "version": "8.4.0-SNAPSHOT"
          },
          "type": "UPGRADE"
        }

Test Case 2

• I tried to pass version "8.4.0-SNAPSHOT":

• POST kbn:/api/fleet/agents/64be65a3-996a-4eea-99f8-d0577540d5a8/upgrade
  {
    "version": "8.4.0-SNAPSHOT"
  }
  

  {"log.level":"info","@timestamp":"2022-07-21T13:29:20.079Z","log.origin":{"file.name":"http/downloader.go","file.line":265},"message":"download progress from https://snapshots.elastic.co/8.4.0-31269fd2/downloads/beats/elastic-agent/elastic-agent-8.4.0-SNAPSHOT-linux-x86_64.tar.gz is 148.5MB/391.1MB (37.97% complete) @ 4.95MBps","ecs.version":"1.6.0"}
  {"log.level":"info","@timestamp":"2022-07-21T13:30:07.871Z","log.origin":{"file.name":"http/downloader.go","file.line":286},"message":"download from https://snapshots.elastic.co/8.4.0-31269fd2/downloads/beats/elastic-agent/elastic-agent-8.4.0-SNAPSHOT-linux-x86_64.tar.gz completed in About a minute @ 5.079MBps","ecs.version":"1.6.0"}
  {"log.level":"info","@timestamp":"2022-07-21T13:30:08.606Z","log.origin":{"file.name":"http/downloader.go","file.line":286},"message":"download from https://snapshots.elastic.co/8.4.0-31269fd2/downloads/beats/elastic-agent/elastic-agent-8.4.0-SNAPSHOT-linux-x86_64.tar.gz.sha512 completed in Less than a second @ +InfYBps","ecs.version":"1.6.0"}
  {"log.level":"error","@timestamp":"2022-07-21T13:30:16.490Z","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2022-07-21T13:30:16Z - message: Application: [e7e294cf-f23e-42f4-a691-72e3866b54c3]: State changed to FAILED: failed to invoke rollback watcher: fork/exec /usr/share/elastic-agent/state/data/data/elastic-agent-0649ad/elastic-agent: no such file or directory - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}
  {"log.level":"error","@timestamp":"2022-07-21T13:30:16.490Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":211},"message":"failed to dispatch actions, error: failed to invoke rollback watcher: fork/exec /usr/share/elastic-agent/state/data/data/elastic-agent-0649ad/elastic-agent: no such file or directory","ecs.version":"1.6.0"}
  {"log.level":"error","@timestamp":"2022-07-21T13:30:16.490Z","log.origin":{"file.name":"status/reporter.go","file.line":260},"message":"Elastic Agent status changed to \"error\": \"component gateway-17ad4b3a: failed to dispatch actions, error: failed to invoke rollback watcher: fork/exec /usr/share/elastic-agent/state/data/data/elastic-agent-0649ad/elastic-agent: no such file or directory\"","ecs.version":"1.6.0"}
  

• Source URI is not there as well and the agent is downloading from a different hardcoded path https://snapshots.elastic.co/

[nimarezainia]

fyi @michalpristas & @pierrehilbert

[michalpristas]

tested with latest snapshot after changing URI i see this in logs: [elastic_agent][info] Source URI changed from "https://artifacts.elastic.co/downloads/" to "https://hello.my.download"

when initiating update to SNAPSHOT, snapshot artifact URI is hit as expected:

{"log.level":"info","@timestamp":"2022-07-29T10:59:23.001Z","log.origin":{"file.name":"http/downloader.go","file.line":265},"message":"download progress from https://snapshots.elastic.co/8.4.0-fb724a71/downloads/beats/elastic-agent/elastic-agent-8.4.0-SNAPSHOT-darwin-x86_64.tar.gz is 169.9MB/272.3MB (62.39% complete) @ 5.664MBps","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-29T10:59:39.654Z","log.origin":{"file.name":"http/downloader.go","file.line":286},"message":"download from https://snapshots.elastic.co/8.4.0-fb724a71/downloads/beats/elastic-agent/elastic-agent-8.4.0-SNAPSHOT-darwin-x86_64.tar.gz completed in 46 seconds @ 5.92MBps","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-29T10:59:39.861Z","log.origin":{"file.name":"http/downloader.go","file.line":286},"message":"download from https://snapshots.elastic.co/8.4.0-fb724a71/downloads/beats/elastic-agent/elastic-agent-8.4.0-SNAPSHOT-darwin-x86_64.tar.gz.sha512 completed in Less than a second @ +InfYBps","ecs.version":"1.6.0"}

when initiating with made up version

--data-raw '{
  "force": true,
  "version": "8.3.14852"
}'

resulted in:

* fetching package failed: Get \"https://hello.my.download/beats/elastic-agent/elastic-agent-8.3.14852-darwin-x86_64.tar.gz\": dial tcp 162.243.62.14:443: connect: operation timed out\n\n - type: 'ERROR' - sub_type: 'FAILED'

[michalpristas]

@criamico have you managed to test it on your end? can we close the issue now?

[criamico]

yes, I think you can close this issue @michalpristas

[michalpristas]

thanks,closing