Remove Beats central management #25696

michel-laterman · 2021-05-12T20:59:22Z

What does this PR do?

Remove the Kibana beats central management feature as it has been
replaced by Fleet. Remove xpack command injection and replace with
an includes mod. Restructure the xpack/libbeat/management module
to include all fleet code in the directory (no api or fleet subdirs).

Checklist

My code follows the style guidelines of this project
~~I have commented my code, particularly in hard-to-understand areas~~
I have made corresponding changes to the documentation
~~I have made corresponding change to the default configuration files~~
~~I have added tests that prove my fix is effective or that my feature works~~
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Run ES 8 SNAPSHOT elastic-package stack up --version=8.0.0-SNAPSHOT -v
Package agent as a snapshot (on macos) cd x-pack/elastic-agent && SNAPSHOT=true DEV=true PLATFORMS=darwin mage package
Untar and enroll the agent using instructions from the Kibana fleet page
start agent and view in ES

Related issues

Closes Remove Beats Central management code #23908

Remove the Kibana beats central management feature as it has been replaced by Fleet.

elasticmachine · 2021-05-12T22:10:30Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: michel-laterman commented: /test
Start Time: 2021-05-28T19:55:03.881+0000
Duration: 60 min 8 sec
Commit: b1e7ccf

Test stats 🧪

Test	Results
Failed	0
Passed	327
Skipped	9
Total	336

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	327
Skipped	9
Total	336

ph · 2021-05-13T19:06:48Z

@jen-huang do you want to synchronize this removal with fleet or once we are green we can merge it?

michel-laterman · 2021-05-13T23:16:17Z

jenkins run tests

michel-laterman · 2021-05-14T16:46:26Z

jenkins run tests

ph · 2021-05-14T18:54:58Z

@michel-laterman Did you rebase with master? I see its the arm CI Job that are failling.

jen-huang · 2021-05-17T23:22:56Z

@ph @michel-laterman Merge at will. On Kibana side we have a draft PR for BCM removal elastic/kibana#99789 already so it should follow closely regardless.

urso · 2021-05-18T11:28:13Z

x-pack/libbeat/management/blacklist.go

@@ -61,7 +60,7 @@ func NewConfigBlacklist(cfg ConfigBlacklistSettings) (*ConfigBlacklist, error) {
 }

 // Detect an error if any of the given config blocks is blacklisted
-func (c *ConfigBlacklist) Detect(configBlocks api.ConfigBlocks) Errors {
+func (c *ConfigBlacklist) Detect(configBlocks ConfigBlocks) Errors {


@ph @blakerouse Can we remove the filtering for output types (and others) from libbeat as well? These have been introduced in the context of Central Management, in order to disallow some potential risky setting.

Agent should disallow the console and file input? Filebeat modules should not be configurable anyways. For customizations we already have capabilities in Agent.
Do I miss something why we should keep 'blacklisting' functionality in libbeat?

I agree we can remove it, but lets keep the PR focus on removing BCM we can do a follow-up to remove the "reject/accept" functionality after we have this merge.

Note this old feature is now unnecessary to keep because of the capabilities of Elastic Agent.

michel-laterman · 2021-05-18T22:40:33Z

x-pack/libbeat/management/config.go

 func defaultConfig() *Config {
 	return &Config{
-		Mode:   ModeCentralManagement,


I don't think this should have many side effects, but it's worth noting that I changed the default mode (also not sure if we should define a Mode attribute and ModeFleet constant anymore

Yeah, I think the Mode can be removed as well.

for this removal please update x-pack/elastic-agent/spec/*.yml then mage update will need to be run.

michel-laterman · 2021-05-18T22:42:00Z

x-pack/libbeat/management/config.go


-	data, err := yaml.Marshal(tmp)
+// ConfigBlocksEqual returns true if the given config blocks are equal, false if not
+func ConfigBlocksEqual(a, b ConfigBlocks) (bool, error) {


I don't think that this is used by fleet, should it also be removed?

If it's not being used, it should be removed. CI/compiler should complain if you removed too much (I hope).

michel-laterman · 2021-05-18T22:42:37Z

x-pack/libbeat/management/error.go

-func (e *Error) EventType() api.EventType {
-	return ErrorEvent
-}
-
 // MarshalJSON transform an error into a JSON document.
 func (e *Error) MarshalJSON() ([]byte, error) {


Are the MarshalJSON/UnmarshalJSON functions used by fleet?

@blakerouse @michalpristas Any idea?

I do not believe they are used.

urso · 2021-05-20T10:17:36Z

Doc build failed with:

INFO:build_docs:Bad cross-document links:
INFO:build_docs:  /tmp/docsbuild/target_repo/html/en/kibana/master/managing-beats.html contains broken links to:
INFO:build_docs:   - en/beats/filebeat/master/configuration-central-management.html
INFO:build_docs:   - en/beats/metricbeat/master/configuration-central-management.html

The reference to the removed docs is in the Kibana repository.

michel-laterman · 2021-05-20T18:06:38Z

Docs will be removed as part of the Kibana PR elastic/kibana#99789

michalpristas · 2021-05-25T08:03:25Z

x-pack/libbeat/management/error.go

@@ -21,58 +17,13 @@ type ErrorType string
 // ConfigError is the type of error send when an unpack or a blacklist happen.
 var ConfigError = ErrorType("CONFIG")

-// ErrorEvent is the event type when an error happen.
-var ErrorEvent = api.EventType("ERROR")
-
 // Error is a config error to be reported to kibana.
 type Error struct {


i winder if we need to keep it here, we are coupling error to type and UUID but Error method just calls Error of nested one and we dont expose coupled information anywhere.

Good point. It looks that the only reason for having this type was json marshalling/unmarshalling. Might indeed be obsolete as well.

elasticmachine · 2021-05-25T18:19:14Z

Pinging @elastic/agent (Team:Agent)

michel-laterman · 2021-05-27T15:35:58Z

/test

urso · 2021-05-28T11:13:10Z

The packaging problems should be fixed in master by now.

michel-laterman · 2021-05-28T19:06:22Z

/test

michel-laterman · 2021-05-28T19:54:42Z

/test

michalpristas · 2021-05-31T09:16:32Z

/package

michalpristas

code looks ok, i think all things were addressed.
Tested ok, standalone and managed

Remove the Kibana beats central management feature as it has been replaced by Fleet. Remove xpack command injection and replace with an includes mod. Restructure the xpack/libbeat/management module to include all fleet code in the directory (no api or fleet subdirs). (cherry picked from commit 27e76c5) # Conflicts: # x-pack/libbeat/include/include.go # x-pack/libbeat/management/api/enroll_test.go # x-pack/winlogbeat/cmd/root.go

- Update to Go 1.16.4 - Adapt to elastic/beats#25696

* Remove Beats central management (#25696) Remove the Kibana beats central management feature as it has been replaced by Fleet. Remove xpack command injection and replace with an includes mod. Restructure the xpack/libbeat/management module to include all fleet code in the directory (no api or fleet subdirs). (cherry picked from commit 27e76c5) # Conflicts: # x-pack/libbeat/include/include.go # x-pack/libbeat/management/api/enroll_test.go # x-pack/winlogbeat/cmd/root.go * Fix cherry-pick Co-authored-by: Michel Laterman <82832767+michel-laterman@users.noreply.github.com> Co-authored-by: michel-laterman <michel.laterman@elastic.co>

* Update to elastic/beats@27e76c567711 - Update to Go 1.16.4 - Adapt to elastic/beats#25696 * golint fixes * tests: chown go.sum files go mod download will download all modules, even if they are not used (e.g. dependencies of parts of beats that are not dependencies of apm-server). As such the command may need to update go.sum, so make sure it is owned by the running user. * Add gotestsum to go.mod

- Update to Go 1.16.4 - Adapt to elastic/beats#25696 - golint fixes - Add gotestsum to go.mod - tests: chown go.sum files go mod download will download all modules, even if they are not used (e.g. dependencies of parts of beats that are not dependencies of apm-server). As such the command may need to update go.sum, so make sure it is owned by the running user.

* Update to elastic/beats@27e76c567711 - Update to Go 1.16.4 - Adapt to elastic/beats#25696 * golint fixes * tests: chown go.sum files go mod download will download all modules, even if they are not used (e.g. dependencies of parts of beats that are not dependencies of apm-server). As such the command may need to update go.sum, so make sure it is owned by the running user. * Add gotestsum to go.mod

michel-laterman added 2 commits May 12, 2021 11:08

Remove beats central management feature.

97b11b9

Remove the Kibana beats central management feature as it has been replaced by Fleet.

Flatten directories so fleet code is directly in management.

834dbff

michel-laterman added cleanup v7.14.0 Team:Elastic-Agent Label for the Agent team backport-v7.14.0 Automated backport with mergify labels May 12, 2021

michel-laterman requested review from ph, urso and michalpristas May 12, 2021 20:59

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels May 12, 2021

Remove central management python integ tests

2a84b53

Merge branch 'master' into remove-central-mgmt

30adf9e

urso reviewed May 18, 2021

View reviewed changes

michel-laterman commented May 18, 2021

View reviewed changes

michel-laterman added 3 commits May 18, 2021 16:07

Add CHANGELOG

8c10a8c

Remove mode attribute and unused code

f235c59

Remove central management from docs

33132a5

michalpristas reviewed May 25, 2021

View reviewed changes

michel-laterman marked this pull request as ready for review May 25, 2021 18:19

michel-laterman requested a review from a team as a code owner May 25, 2021 18:19

Merge branch 'master' into remove-central-mgmt

7d037e9

Merge branch 'master' into remove-central-mgmt

b1e7ccf

michalpristas approved these changes May 31, 2021

View reviewed changes

urso approved these changes May 31, 2021

View reviewed changes

michel-laterman merged commit 27e76c5 into elastic:master May 31, 2021

michel-laterman deleted the remove-central-mgmt branch May 31, 2021 14:44

mergify bot mentioned this pull request May 31, 2021

[7.x](backport #25696) Remove Beats central management #26012

Merged

axw added a commit to axw/apm-server that referenced this pull request Jun 1, 2021

Update to elastic/beats@27e76c567711

e37aa5c

- Update to Go 1.16.4 - Adapt to elastic/beats#25696

axw added a commit to axw/apm-server that referenced this pull request Jun 1, 2021

Update to elastic/beats@27e76c567711

1c43b1b

- Update to Go 1.16.4 - Adapt to elastic/beats#25696

axw mentioned this pull request Jun 1, 2021

Update to elastic/beats@27e76c567711 elastic/apm-server#5382

Merged

axw mentioned this pull request Jun 2, 2021

[7.x] Update to elastic/beats@d95f6bab8554 elastic/apm-server#5397

Merged

ycombinator mentioned this pull request Jun 2, 2021

Apache Status metricset is no longer adjusting fields for fleet mode #26120

Closed

adriansr mentioned this pull request Jun 15, 2021

Tests are failing for the x-pack/packetbeat help enroll elastic/beats-tester#195

Closed

This was referenced Jun 15, 2021

Revert #105 ChrsMark/beats-tester#1

Closed

Revert #105 elastic/beats-tester#196

Merged

adriansr mentioned this pull request Jun 15, 2021

Add tests to ensure a Beat binary belongs to the correct distribution (Elastic or oss) elastic/beats-tester#197

Open

aleksmaus mentioned this pull request Jun 15, 2021

[7.13](backport #26324) Osquerybeat: Align with the rest of the beats, set the ECS version #26333

Closed

This was referenced Aug 11, 2021

Remove code related to Beats Central Management elastic/elasticsearch#76330

Open

[Monitoring] Remove code related to Beats Central Management elastic/kibana#108137

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove Beats central management #25696

Remove Beats central management #25696

michel-laterman commented May 12, 2021 •

edited

Loading

elasticmachine commented May 12, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

ph commented May 13, 2021

michel-laterman commented May 13, 2021

michel-laterman commented May 14, 2021

ph commented May 14, 2021

jen-huang commented May 17, 2021

urso May 18, 2021

ph May 18, 2021

michel-laterman May 18, 2021

urso May 19, 2021

michel-laterman May 19, 2021

michalpristas May 25, 2021

michel-laterman May 26, 2021

michel-laterman May 18, 2021

urso May 19, 2021

michel-laterman May 18, 2021

urso May 19, 2021

blakerouse May 19, 2021

urso commented May 20, 2021 •

edited

Loading

michel-laterman commented May 20, 2021

michalpristas May 25, 2021

urso May 25, 2021

elasticmachine commented May 25, 2021

michel-laterman commented May 27, 2021

urso commented May 28, 2021

michel-laterman commented May 28, 2021

michel-laterman commented May 28, 2021

michalpristas commented May 31, 2021

michalpristas left a comment

Remove Beats central management #25696

Remove Beats central management #25696

Conversation

michel-laterman commented May 12, 2021 • edited Loading

What does this PR do?

Checklist

How to test this PR locally

Related issues

elasticmachine commented May 12, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

ph commented May 13, 2021

michel-laterman commented May 13, 2021

michel-laterman commented May 14, 2021

ph commented May 14, 2021

jen-huang commented May 17, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

urso commented May 20, 2021 • edited Loading

michel-laterman commented May 20, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented May 25, 2021

michel-laterman commented May 27, 2021

urso commented May 28, 2021

michel-laterman commented May 28, 2021

michel-laterman commented May 28, 2021

michalpristas commented May 31, 2021

michalpristas left a comment

Choose a reason for hiding this comment

michel-laterman commented May 12, 2021 •

edited

Loading

elasticmachine commented May 12, 2021 •

edited by jenkins-beats-ci bot

Loading

urso commented May 20, 2021 •

edited

Loading