[Fleet] Remove default integrations

[mostlyjason]

Currently, the system, elastic agent, and fleet server packages are automatically installed. The system integration is also added to the default agent policy in Fleet. This creates problems for the onboarding flow in other solutions and products #82851. It's also not the best UX because we are adding these integrations without explicit consent from the user.

What integrations should be installed by default, if any? If not, when should they get installed? How should this work in self-managed clusters, and on cloud where the APM & Fleet node is added by default? How should it work for standalone agents?

In particular, the system integration should be more explicit. That also means that we won't install the integration by default but instead when the user takes action to install it. It'd be still be nice to encourage users to add these packages as a useful way to get started or monitor hosts, but it should still be their choice. Additionally, it's not obvious that the system integration is the one users should install since we have separate integrations for linux and windows. Some proactive prompting should help users get started.

Potential places to add the system integration include when the user is adding their first agent in Fleet (potentially creating their first agent policy instead of using a default), adding their first integration (should we add it here or not), and creating a new agent policy.

Here it is not explicit choice:

Here it is an explicit choice:

Also, users are prevented from removing or reininstalling these default packages https://discuss.elastic.co/t/reinstall-system-integration-assets/283140. If we remove their special status, then users should have full control over them.

Another use case to consider is if the user adds the system integration from the integrations app. In this case, the user probably does not intend to add a duplicate integration policy. We might not to recommend adding the system integration to a new agent policy, if the user is already adding one.

UI automation tests: #121436

Related:

• https://github.com/elastic/sdh-beats/issues/1217
• [7.16] [Fleet] Enable auto-upgrades for Stack-aligned packages during Fleet setup #112831
• https://github.com/elastic/observability-test-environments/issues/2030
• Remove dependency on Fleet Default policy e2e-testing#2039
• [8.1.0-SNAPSHOT] Fleet Server can't enroll: FAILED: Missed two check-ins fleet-server#1129 (comment)
• [automation] update elastic stack version for testing 8.2.0-5d69c4c3 apm-server#7227
• Fix setup with Elastic Agent and Fleet apm-integration-testing#1440

Blockers:

• Predefine agent policies in the Kibana config elastic-package#655
• Remove dependency on Fleet Default policy cloud-on-k8s#5262
• Add dedicated policy for agents started through the vm extension azure-vm-extension#11

CC @dborodyansky @mukeshelastic

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[alexfrancoeur]

Just wanted to share an experience I had with a 7.16 snapshot in Cloud staging. For quick demos, I typically add a system integration, choose the default config and run with it. This is the first time I've received this error. I'm sure it's intentional, but I'd imagine many folks will just click through when onboarding to Elastic for the first time.\

There is already a package with the same name on this agent policy is the error received. First, I don't believe we reference packages anywhere anymore. For a new user, this error doesn't make much sense. Second, if this is in fact primarily an issue with duplicate names or ID's, can the system integration that's automatically installed with Kibana simply have a different name than the default value? Not sure if this is a quick fix, but if so, it might make sense to update for 7.16 as I imagine this integration could be used more.

[kpollich]

@alexfrancoeur this is a regression we captured in #116475. We have a PR close to landing that fixes the regressions, as well as guarantees globally unique policy names here: #115212

[alexfrancoeur]

Awesome, thanks for the update @kpollich !

[juliaElastic]

@mostlyjason @joshdover I went through the issue and comments in linked issue. Here is my analysis so far, let me know your thoughts.

Current state after Fleet setup:

• Default policy has system integration by default (more than 100 assets)
• Default fleet server policy / Elastic Cloud agent policy has fleet-server integration by default (no assets)
• elastic-agent package is installed, not assigned to agent policy (6 assets)

Proposal:

Phase 1: Default policy with system package and elastic-agent package
Both on-prem and cloud:

• Do not create Default policy on setup

• Add Agent flyout: when no agent policies exist, add a link to Create agent policy flyout

  • Missing from design doc

• Add integration: see improvements in design doc (New hosts/Existing hosts)

• Create Agent policy flyout: see improvements in design doc

• elastic-agent package: do not install by default, only with first agent policy, see above

Phase 2: Default fleet server policy with fleet-server package

• On prem:

  • Do not create Default fleet server policy by default
  • On-prem instructions: see improvements in design doc

• Cloud: should we keep Elastic Cloud agent policy as is? Since Fleet Server is managed by cloud, does it make sense to delay the setup?

• Confirmed that cloud flow shouldn't change, we should make sure to install fleet-server integration on cloud by default (logic might have to move since currently this is done by fleet startup)

Additional tasks:

• Make default packages removable

EDIT: saw the UX design doc, commented a few things there.
Overall the proposed design makes sense to me.

[mostlyjason]

@juliaElastic I have some questions about your task list. I'll schedule a meeting to review it with Dmitry.

[juliaElastic]

@mostlyjason I added my comment before seeing the design initially, updated now to reflect my understanding after seeing the designs.

[kuisathaverat]

Is there a way to have the YAML equivalent to the UI configuration? for example to APM integration

I've started to configure this and make the use of fleet more complicated than it is, IMHO to add a policy if you going to use the default settings is complicated to justify, for example, if you want to add an APM integration this can be the configuration

xpack.fleet.agents.enabled: true
xpack.fleet.packages:
    - name: system
      version: latest
    - name: elastic_agent
      version: latest
    - name: apm
      version: latest
    - name: fleet_server
      version: latest
xpack.fleet.agentPolicies:
    - name: Apm Agent policy
      id: apm-agent-policy
      description: Agent policy with APM and System logs and metrics enabled
      is_managed: false
      namespace: default
      monitoring_enabled:
          - logs
          - metrics
      package_policies:
          - name: system-1
            id: default-system
            package:
                name: system
          - name: apm-1
            id: default-apm
            package:
                name: apm

            <-- Here, you have to add all the APM configuration that I do not have any idea how it looks in YAML but looks like a long piece of YAML -->

    - name: Fleet Server policy preconfigured
      id: fleet-server-policy
      namespace: default
      package_policies:
          - name: Fleet Server
            package:
                name: fleet_server
            inputs:
                - type: fleet-server
                  keep_enabled: true
                  vars:
                      - name: host
                        value: 0.0.0.0
                        frozen: true
                      - name: port
                        value: 8220
                        frozen: true
    - name: Fleet Server policy preconfigured
      id: fleet-server-policy
      namespace: default
      package_policies:
          - name: Fleet Server
            package:
                name: fleet_server
            inputs:
                - type: fleet-server
                  keep_enabled: true
                  vars:
                      - name: host
                        value: 0.0.0.0
                        frozen: true
                      - name: port
                        value: 8220
                        frozen: true

Can we simplify the configuration allowing to choose to create the default policy? by adding a couple of parameters (one required the other optional) we can create those policies, and the configuration is much easier.

xpack.fleet.agents.enabled: true
xpack.fleet.packages:
    - name: system
      version: latest
    - name: elastic_agent
      version: latest
    - name: apm
      version: latest
      create_default: 
        name: APM default policy
        id: apm-default-policy
    - name: fleet_server
      version: latest
      create_default: 
        name: Fleet Server default policy
        id: fllet-server-default-policy

Other alternative is to have a configuration generator in the UI that allows you to create a policy with all integration you need, then generate the YAML, then I only have to copy and paste in the Kibana.yml file, it will be a long piece of YAML but at least I do not have to create it manually. It will continue to be hard to read, edit, and without a way to validate that the syntax is correct other than launch Kibana that is not a quick operation.

[joshdover]

Can we simplify the configuration allowing to choose to create the default policy? by adding a couple of parameters (one required the other optional) we can create those policies, and the configuration is much easier.

This feels like a pretty reasonable request and would mirror the improvements we did on the API in #119739. @juliaElastic WDYT, should we open an issue for this?

[juliaElastic]

@kuisathaverat Good point that adding the defaults to preconfig is not that simple.
How is the APM integration added currently? The Fleet change only removes the default policies, APM was not included in defaults before.
I like the idea of create_default in preconfig (we need a way to distinguish normal policy from fleet server policy), should we raise this for 8.2?

[joshdover]

How is the APM integration added currently? The Fleet change only removes the default policies, APM was not included in defaults before.

This is a good question, I don't see how this change impacts setting up any APM policies.

I like the idea of create_default in preconfig (we need a way to distinguish normal policy from fleet server policy), should we raise this for 8.2?

We'll have to see on timing, but I think an issue to start collecting use cases would be helpful.

[juliaElastic]

Created this: #124030
We should be careful with the naming, since we want to move away from the concept of default policies.

[kuisathaverat]

How is the APM integration added currently? The Fleet change only removes the default policies, APM was not included in defaults before.

This is a good question, I don't see how this change impacts setting up any APM policies.

To start the APM server now you use, and Elastic Agent and the APM integration, it is true that the current default policy does not have the APM integration. To have a way to create a policy for an integration with the default values in a single config step simplifies the configuration. It is not directly related to remove the default policies but it could reuse the work in #124030
The use case could be something like:

• GIVEN a user wants to connect an APM agent
• AND an Elastic stack cluster up and running
• AND a Fleet Server up and running
• WHEN adding an APM package to Kibana creates a APM policy with the default values

xpack.fleet.packages:
    - name: elastic_agent
      version: latest
    - name: apm
      version: latest
      create_default: 
        name: APM default policy
        id: apm-default-policy

• THEN a new policy is created with id apm-default-policy ready to use with an Elastic Agent

I am thinking on environments like our test environments that are configured as code for everything.

[juliaElastic]

I see, so this use case is similar to what we added on UI to create a new agent policy in Add integration flow.

I think we have to consider that users may want to add multiple packages to an agent policy with preconfig (e.g apm and system). It could be represented as something like this:

xpack.fleet.agentPolicies:
  - name: Agent policy 1
    id: agent-policy-1
    add_packages: system, apm 

[nchaulet]

Instead of introducing a new concept can we allow the actual preconfiguration to use default inputs like we now do in the package policy API

xpack.fleet.agentPolicies:
    - name: Apm Agent policy
      id: apm-agent-policy
      description: Agent policy with APM and System logs and metrics enabled
      is_managed: false
      namespace: default
      monitoring_enabled:
          - logs
          - metrics
      package_policies:
          - name: system-1
            id: default-system
            package:
                name: system
          - name: apm-1
            id: default-apm
            package:
                name: apm

[juliaElastic]

that's a good point, package_policies inputs are optional, and come with default values

[joshdover]

Thanks all for the feedback here. Definitely agree that using the default values for policy inputs makes sense. Let's move this discussion over to the issue that Julia created (#124030) to avoid confusing folks who are following this issue which is only tangentially related.

[juliaElastic]

@joshdover @jen-huang @mostlyjason We got confirmation today about the blockers, and it seems we are good to go. Are you okay with merging this feature then for 8.1? The remaining dependencies can be finished after FF.

• Azure VM: POC working, they are ok with the timeline
• elastic-package: pr working with preconfiguration, should be ok
• ECK: documentation update is raised as a pr, should be resolved soon
• e2e-testing: it seems as no impact, waiting for confirmation
• observability-test-environments: preconfiguration POC in progress, should be ok

[joshdover]

@juliaElastic I'm comfortable moving forward with merging this for 8.1 based on those statuses. Let's be sure to be available to support any teams that need any additional help, but I know you've already been super on top of that 😄

[juliaElastic]

@dikshachauhan-qasource @amolnater-qasource FYI this feature has been merged for 8.1, it might impact some of your existing manual testcases

[jen-huang]

Thanks for pushing this over the finish line @juliaElastic. Can we close this and track pending items in another issue, if needed?

[juliaElastic]

Thanks for pushing this over the finish line @juliaElastic. Can we close this and track pending items in another issue, if needed?

@jen-huang yes, I think we can close it

[amolnater-qasource]

Hi @joshdover & @juliaElastic
We have proposed below testcases for #108456

• [Self-managed]Observe no Integration and policy is preinstalled on 8.1 Self-managed environment.
• [Self-managed]Observe on the Agents tab, user is able to create a new policy on a fresh setup self-managed environment.
• [Self-managed]Observe on the Agents tab, clicking Add Agent button, user is able to create a new policy on a fresh setup self-managed environment.
• [Self-managed]Observe new policy created from the Agents tab on a self-managed setup installs System & Fleet Server integration.
• Observe that only Elastic cloud agent policy is available on cloud.
• Observe that on the cloud only Fleet server and APM integration is installed by default.
• Observe on the cloud System and Elastic Agent integration is not installed by default.
• While creating a new policy, it installs System and Elastic Agent integration.

Could you please review these and let us know if we are missing anything?
Further could you please share any specific feature ticket for below feature(if any):

Thanks

[juliaElastic]

Hi @amolnater-qasource, thank you for this, a few comments:

• [Self-managed]Observe on the Agents tab, user is able to create a new policy on a fresh setup self-managed environment.

I would add "user is able to create a new Fleet Server agent policy"

• [Self-managed]Observe on the Agents tab, clicking Add Agent button, user is able to create a new policy on a fresh setup self-managed environment.

• Enroll in Fleet tab: user is able to create a new Fleet Server agent policy.
  • After Fleet Server is enrolled, user is able to create a new Agent policy on this tab
• Run standalone tab: user is able to create a new Agent policy
• User is able to enroll Fleet Server using the newly created Fleet Server agent policy
• User is able to enroll Elastic Agent using the newly created Agent policy

• [Self-managed]Observe new policy created from the Agents tab on a self-managed setup installs System & Fleet Server integration.

It also installs Elastic Agent integration (without integration policy)

Could you please review these and let us know if we are missing anything? Further could you please share any specific feature ticket for below feature(if any):

The feature of New/Existing hosts on Add integration form is included in the same ticket (details are in UX design doc).
See this in description:
"Potential places to add the system integration include when the user is adding their first agent in Fleet (potentially creating their first agent policy instead of using a default), adding their first integration (should we add it here or not), and creating a new agent policy."

A few more testcases:

• User is able to Add integration with a new or existing integration policy
• User is able to delete all agent policies on Agent policy details page > Settings tab > Delete policy button
• User is able to delete all integration policies (including system, elastic_agent, fleet_server) on Integration details page > Integration policies tab > Delete integration
• If exactly one agent policy exists, it should be preselected on agent policy dropdown (Add agent flyout, Add integration > Existing hosts)
• If more than 1 agent policies exist, no agent policy should be preselected in dropdown (Add agent flyout, Add integration > Existing hosts)

[amolnater-qasource]

Hi @juliaElastic
Thanks for providing detailed updates on test content, we have updated our doc with proposed testcases for this feature. (Please lemme know in case of any permission issues for accessibility/editing this doc.)

Further we have also added few testcases related to enrollment tokens.

We will be testing this feature more and will be creating more scenarios, if required.

Thanks

[amolnater-qasource]

Hi @juliaElastic
We have create 23 testcases in our ingest manager test suite at link

Please let us know if we are missing anything.

Thanks