[Fleet] Resolve latest package version from bundled packages if possible

[kpollich]

Summary

Fixes #126487

Update "fetch latest package" logic to compare any bundled packages on disk to the version returned from the registry. If the bundled package version is newer, treat that as the latest.

To test

Create a mimicked cloud policy via kibana.dev.yml

xpack.fleet.packages:
  - name: fleet_server
    version: latest
  - name: apm
    version: latest

xpack.fleet.agentPolicies:
  # Cloud Agent policy
  - name: Elastic Cloud agent policy
    description: Default agent policy for agents hosted on Elastic Cloud
    id: policy-elastic-agent-on-cloud

    is_default: false
    is_managed: true
    is_default_fleet_server: false

    namespace: default
    monitoring_enabled: []
    # If a user scales up / down the Elastic Agent in Cloud, the old
    # instances will still be shown as offline for 24h
    # The reason for having a value such high is to prevent fleet-servers to self unenroll to soon
    # in case checkin cannot happen for a longer time period.
    unenroll_timeout: 86400 # 1 day TTL

    package_policies:
      - name: Fleet Server
        id: elastic-cloud-fleet-server
        package:
          name: fleet_server
        inputs:
          - type: fleet-server
            keep_enabled: true

            vars:
              - name: host
                value: 0.0.0.0
                frozen: true
              - name: port
                value: 8220
                frozen: true
              - name: custom
                value: |
                  server.runtime:
                    gc_percent: 20          # Force the GC to execute more frequently: see https://golang.org/pkg/runtime/debug/#SetGCPercent
      - name: Elastic APM
        id: elastic-cloud-apm
        package:
          name: apm
        inputs:
          - type: apm
            keep_enabled: true
            vars:
              - name: api_key_enabled
                value: true
              - name: host
                value: '0.0.0.0:8200'
                frozen: true
              - name: secret_token
                value: secret
              - name: tls_enabled
                value: true
                frozen: true
              - name: tls_certificate
                value: /app/config/certs/node.crt
                frozen: true
              - name: tls_key
                value: /app/config/certs/node.key
                frozen: true
              - name: url
                value: http://example.com
                frozen: true

Then, update your snapshot URL to always resolve to production instead of snapshot/staging by editing this file:

https://github.com/elastic/kibana/blob/4b1bf83bad93b2349581ac7ef70adf230d4dfdb1/x-pack/plugins/fleet/server/services/epm/registry/registry_url.ts

Then, run Fleet setup in a fresh ES/Kibana environment. Verify that APM 8.1.0 and Fleet Server 1.1.0 are installed as expected. e.g.

[2022-02-28T13:10:55.793-05:00][DEBUG][plugins.fleet] kicking off bulk install of fleet_server, apm
[2022-02-28T13:10:55.799-05:00][DEBUG][plugins.fleet] found bundled package for requested install of apm-8.1.0 - installing from bundled package archive
[2022-02-28T13:10:55.800-05:00][DEBUG][plugins.fleet] found bundled package for requested install of fleet_server-1.1.0 - installing from bundled package archive

Note: This does result in a rather broken UI experience on the integration details page, because the installed version does not exist in the configured registry, e.g

However, this is expected for now, and the policy editor for this version of APM is still functional.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[kpollich]

These fetch functions could really use some unit tests at this point now that there are varying fallback cases, etc.

[joshdover]

Good idea, I added an item in our board 👍

[joshdover]

Tested locally, policy editor worked 🎉

As you noted, the Integration details UI is broken now, which is sorta different behavior for testers but at least this unblocks the testing of APM server on Cloud. Once package is published to prod, this will resolve.

One semi-related thing I noticed was the link on the APM server tutorial to Fleet goes to the 8.0.0 page. It doesn't break anything though, everything on the page says 8.1.0 and the editor works fine.

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 2ba1dfb
• Storybooks Preview
• Webpack Bundle Analyzer

Metrics [docs]

✅ unchanged

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @kpollich

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	8.1	Backport failed because of merge conflicts

How to fix

Re-run the backport manually:

node scripts/backport --pr 126492

Questions ?

Please refer to the Backport tool documentation

[kpollich]

Waiting on a backport of a previous PR to 8.1 before I can resolve conflicts reasonably and backport this to 8.1.

#126523

[kpollich]

And the above PR, it turns out needs to wait on a backport of #125959 to 8.1 as well. 🙃