Integrations always running regardless of enabled flag #689

mattdurham · 2021-06-25T13:24:19Z

PR Description

The manager is ALWAYS collecting the integration even if enabled is set to false.

Which issue(s) this PR fixes

Honor the enabled flag, when the manager was changed this behavior looks to have been lost.

Closes #686

Notes to the Reviewer

PR Checklist

[ X] CHANGELOG updated

rfratto · 2021-06-25T13:36:03Z

pkg/integrations/manager.go

@@ -280,6 +280,11 @@ func (m *Manager) ApplyConfig(cfg ManagerConfig) error {
 			shouldCollect = *common.ScrapeIntegration
 		}

+		// If Enabled is false then we should override the should collect
+		if p.cfg.CommonConfig().Enabled == false {


Hmm, I don't think this is being done in the right place. The loop on line 210 would still spawn the integration, which it probably shouldn't be doing.

One way to handle this might be to add a function to return the list of only enabled integrations and iterate over that instead when calling ApplyConfig. That way disabling an integration and removing it from the file are treated as the same.

My thought on that was looking at the documentation is that enabled flag enables the scraping but if you include the integration in the configuration with enabled = false it still will get the statistics but won't send them. Should there be a difference in disabling an integration vs removing it, if the answer is no then I will put in my original fix. Specifically with the agent integration having it available but not sending for debugging purposes seemed useful. Can def go either way though.

Nah, the intent is that enabled drives whether the integration should be ran (so disabling it is the same as removing it), and scrape_integration determines if that running integration should be scraped by the Agent automatically.

rfratto

Nice catch!

…t the config reloader so that it propagates from the source downstream

rfratto · 2021-06-25T14:34:43Z

cmd/agent/main.go

+		// Iterate over the configurations and remove any that Enabled = false, this works by setting i to the last element
+		//   and then returning the array minus the last element. This loses order but is performant.
+		for i := 0; i < len(cfg.Integrations.Integrations);{
+			if !cfg.Integrations.Integrations[i].CommonConfig().Enabled {
+				cfg.Integrations.Integrations[i] = cfg.Integrations.Integrations[len(cfg.Integrations.Integrations)-1]
+				cfg.Integrations.Integrations = cfg.Integrations.Integrations[:len(cfg.Integrations.Integrations)-1]
+			} else {
+				i++
+			}
+		}


Can you create a new function and call it from integrations.Manager.ApplyConfig instead? Filtering the actual slice here will omit the config from the results of /-/config which might be a little confusing to users.

This got more complicated. When you remove a configuration it did not stop the scraper, so when a configuration is disabled/removed and reload called Agent would list scrape_error = 404. Since the integration had stopped but the scraper was still running.

mattdurham · 2021-06-25T15:28:51Z

pkg/integrations/manager.go

@@ -220,9 +222,14 @@ func (m *Manager) ApplyConfig(cfg ManagerConfig) error {
 				continue
 			}
 			p.stop()
+			m.im.DeleteConfig(key)


Beforehand the scraper was not stopping, but the integration was.

I'm not following how this is necessary, the second loop should be deleting the config?

With the addition of the enabled check around line 227 the integration process is not started so the integration isn't added to the to m.integrations.

rfratto · 2021-06-25T15:32:19Z

pkg/integrations/manager.go

@@ -268,6 +279,7 @@ func (m *Manager) ApplyConfig(cfg ManagerConfig) error {

 		_ = m.im.DeleteConfig(key)
 		process.stop()
+		m.im.DeleteConfig(key)


Why do we need to call this twice?

Not at all and removed.

rfratto · 2021-06-25T15:37:09Z

pkg/integrations/manager.go

+		if !ic.CommonConfig().Enabled {
+			continue
+		}


I still think a function to return a slice of enabled integrations would be easier to follow than having to maintain this check across the two loops, IMO.

We still need the integrations in the array to disable/remove them, ie going from enabled -> disabled on reload.

rfratto · 2021-06-25T15:37:34Z

pkg/integrations/manager.go

@@ -220,9 +222,14 @@ func (m *Manager) ApplyConfig(cfg ManagerConfig) error {
 				continue
 			}
 			p.stop()
+			m.im.DeleteConfig(key)


I'm not following how this is necessary, the second loop should be deleting the config?

rfratto · 2021-06-25T15:38:34Z

Does this fix #686? That reports that set_collectors doesn't appear to be working with node_exporter, alongside this bug.

mattdurham · 2021-06-25T18:20:20Z

It fixes the enable issue that cropped during #686 , I was unable to reproduce an actual issue with set collectors. I tested several variations and they behaved as expected.

rfratto

Thanks! Can you add a regression test for this?

I'd also like to see a test that provides we need m.im.DeleteConfig in there twice, because it definitely seems unnecessary which makes it a candidate for accidental removal. Having a test would help prevent this.

tbrandsma-rw · 2021-07-20T00:20:45Z

Any updates on when this fix will be merged in? I am currently dealing with the fallout of this issue because the enabled flag isn't being honored.

…so that it generated yaml.

mattdurham · 2021-07-21T15:11:04Z

Updated with unit tests and explanation below. Also had to update the mock integration so marshaling would work in the comparison. No tests were broken.

Situation removal of config on line 225 in manager.go

Config is ENABLED
Integrations and Configs and Processes are all ENABLED
1. integration process is started
2. instance process is started
3. configs added
ApplyConfig is called with integration DISABLED
1. Enter loop at 210
2. exist check succeeds
3. yaml is different
4. integration process is stopped
5. integration is removed from manager integrations
6. config is DISABLED so line 229 moves to next integration
7. loop over manager integrations is called
8. since the integration was removed in the above the DeleteConfig is never called
9. the instance process continues running

If you run TestManager_IntegrationEnabledToDisabledReload with line 225 commented out, then the TestManager_IntegrationEnabledToDisabledReload will fail.

mattdurham · 2021-07-21T15:35:13Z

As far as a release, currently unknown, it is going to be bundled with some other large changes.

rfratto

LGTM after removing the duplicated if statement in the first loop

rfratto · 2021-07-21T17:16:06Z

pkg/integrations/manager.go

+		if !ic.CommonConfig().Enabled {
+			continue
+		}


Looks like this is in here twice :)

* Enabled flag should drive collection * Satisfy linter * Fix mock integration so it sets enabled = true by default * Make enabled the equivalent of not being in the config, removing it at the config reloader so that it propagates from the source downstream * Fix issue where removing the config/disabling did NOT stop the scraper * Remove whitespace * remove unneeded comment * Remove duplicated delete config * Update comments and add unit tests. Also had to fix mock integration so that it generated yaml. * Simplify unit test * change case because of refactor * filter integrations to enabled set * Move check up to the first statement * fix merge * Fix for double loop Co-authored-by: Robert Fratto <robertfratto@gmail.com>

mattdurham added 2 commits June 25, 2021 09:20

Enabled flag should drive collection

c9e476b

Satisfy linter

afa2eee

rfratto reviewed Jun 25, 2021

View reviewed changes

mattdurham added 2 commits June 25, 2021 09:55

Fix mock integration so it sets enabled = true by default

cce8999

Make enabled the equivalent of not being in the config, removing it a…

6808da7

…t the config reloader so that it propagates from the source downstream

rfratto reviewed Jun 25, 2021

View reviewed changes

rfratto changed the title ~~Enabled flag should drive collection~~ Integrations always running regardless of enabled flag Jun 25, 2021

mattdurham added 3 commits June 25, 2021 11:24

Fix issue where removing the config/disabling did NOT stop the scraper

22f1d16

Remove whitespace

cd058b4

remove unneeded comment

e7b90a7

mattdurham commented Jun 25, 2021

View reviewed changes

rfratto reviewed Jun 25, 2021

View reviewed changes

Remove duplicated delete config

0f9ca9c

rfratto reviewed Jul 7, 2021

View reviewed changes

mattdurham added 2 commits July 21, 2021 11:00

Update comments and add unit tests. Also had to fix mock integration …

bb362da

…so that it generated yaml.

Simplify unit test

d752a4f

mattdurham added 2 commits July 21, 2021 11:12

change case because of refactor

b7f0e68

Merge remote-tracking branch 'origin/main' into enabled_issue

a60942a

mattdurham requested a review from rfratto July 21, 2021 15:13

filter integrations to enabled set

98895c0

mattdurham added 3 commits July 21, 2021 12:04

Move check up to the first statement

8c4c896

merge

f90f44a

fix merge

7b1987f

rfratto approved these changes Jul 21, 2021

View reviewed changes

Fix for double loop

411ccde

mattdurham merged commit 1c28f09 into main Jul 21, 2021

mattdurham deleted the enabled_issue branch July 21, 2021 17:55

mattdurham mentioned this pull request Sep 7, 2021

crow doc rfratto/agent#8

Closed

3 tasks

github-actions bot added the frozen-due-to-age Locked due to a period of inactivity. Please open new issues or PRs if more discussion is needed. label Apr 11, 2024

github-actions bot locked as resolved and limited conversation to collaborators Apr 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrations always running regardless of enabled flag #689

Integrations always running regardless of enabled flag #689

mattdurham commented Jun 25, 2021 •

edited

Loading

rfratto Jun 25, 2021 •

edited

Loading

mattdurham Jun 25, 2021 •

edited

Loading

rfratto Jun 25, 2021

rfratto left a comment

rfratto Jun 25, 2021

mattdurham Jun 25, 2021

mattdurham Jun 25, 2021

rfratto Jun 25, 2021

mattdurham Jun 25, 2021

rfratto Jun 25, 2021

mattdurham Jun 25, 2021

rfratto Jun 25, 2021

mattdurham Jun 25, 2021

rfratto Jun 25, 2021

rfratto commented Jun 25, 2021 •

edited

Loading

mattdurham commented Jun 25, 2021

rfratto left a comment

tbrandsma-rw commented Jul 20, 2021

mattdurham commented Jul 21, 2021

mattdurham commented Jul 21, 2021

rfratto left a comment

rfratto Jul 21, 2021

Integrations always running regardless of enabled flag #689

Integrations always running regardless of enabled flag #689

Conversation

mattdurham commented Jun 25, 2021 • edited Loading

PR Description

Which issue(s) this PR fixes

Notes to the Reviewer

PR Checklist

rfratto Jun 25, 2021 • edited Loading

Choose a reason for hiding this comment

mattdurham Jun 25, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rfratto left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rfratto commented Jun 25, 2021 • edited Loading

mattdurham commented Jun 25, 2021

rfratto left a comment

Choose a reason for hiding this comment

tbrandsma-rw commented Jul 20, 2021

mattdurham commented Jul 21, 2021

mattdurham commented Jul 21, 2021

rfratto left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattdurham commented Jun 25, 2021 •

edited

Loading

rfratto Jun 25, 2021 •

edited

Loading

mattdurham Jun 25, 2021 •

edited

Loading

rfratto commented Jun 25, 2021 •

edited

Loading