allow dynamic updating of instance scrape_configs and remote_write #145
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit introduces an Update method against the Instance that allows for applying new changes found on remote_write and scrape_configs sections for a particular instance. If any other field changes, Update will return ErrInvalidUpdate, which will cause the InstanceManager to do a normal "restart" of the instance. Future commits may loosen the restriction on what is able to be dynamically updated. Some code cleanup can be found as part of this commit: fakes that have been called mocks have been updated to use their proper name.
rfratto
commented
Aug 5, 2020
| // 4. Validates that after 15 seconds, the scrape endpoint and remote_write | ||
| // endpoint has been called. | ||
| func TestInstance_Update(t *testing.T) { | ||
| prometheus.DefaultRegisterer = prometheus.NewRegistry() |
There was a problem hiding this comment.
I'm not happy that I have to do this, and I'm planning on cleaning this up with the PR for #141.
rfratto
referenced
this pull request
in rfratto/agent
Aug 5, 2020
This commit migrates InstanceManager to instance.BasicManager. This is to reduce duplication as we add more implementations of the instance.Manager interface (including removing ha.InstanceManager, which was a duplicate of that interface). While creating this commit, I noticed that there was a leaked locked mutex when DeleteConfig fails. That has been fixed with this commit. Related to the work for #141. Depends on #145.
rfratto
commented
Aug 6, 2020
rfratto
commented
Aug 6, 2020
| // failed, indicating that the instance config changed in an incompatible way | ||
| // and it must be fully stopped and replaced with a new instance for the new | ||
| // config to apply. | ||
| func (im *InstanceManager) forceApply(c instance.Config) error { |
There was a problem hiding this comment.
I don't think this method needs to exist. I'd remove it here, but I already took it out with #146
jeschkies
reviewed
Aug 6, 2020
pkg/prom/instance/instance.go
Outdated
| func (i *Instance) Config() Config { | ||
| i.cfgMutex.Lock() | ||
| defer i.cfgMutex.Unlock() | ||
| return i.cfg |
There was a problem hiding this comment.
Just a naive question. Why is the lock required here? Isn't cfg a reference?
There was a problem hiding this comment.
i.cfg will be copied by value here, but the mutex is needed to prevent i.cfg being read in the middle of a write from another Goroutine (i.e., the one calling Update), which is undefined behavior in Go.
Any feedback is appreciated :) Instance is a (poor) name I came up with to describe the combination of the Prometheus components we run. It's the set of service discovery, scraping, remote_write, and a WAL. The Agent can run any number of instances depending on how it's configured. |
gotjosh
approved these changes
Aug 6, 2020
rfratto
referenced
this pull request
in rfratto/agent
Aug 6, 2020
This commit migrates InstanceManager to instance.BasicManager. This is to reduce duplication as we add more implementations of the instance.Manager interface (including removing ha.InstanceManager, which was a duplicate of that interface). While creating this commit, I noticed that there was a leaked locked mutex when DeleteConfig fails. That has been fixed with this commit. Related to the work for #141. Depends on #145.
rfratto
added a commit
that referenced
this pull request
Aug 7, 2020
* move InstanceManager to instance package This commit migrates InstanceManager to instance.BasicManager. This is to reduce duplication as we add more implementations of the instance.Manager interface (including removing ha.InstanceManager, which was a duplicate of that interface). While creating this commit, I noticed that there was a leaked locked mutex when DeleteConfig fails. That has been fixed with this commit. Related to the work for #141. Depends on #145. * Update pkg/prom/instance/manager.go Co-authored-by: gotjosh <josue.abreu@gmail.com> * Update pkg/prom/instance/manager.go Co-authored-by: gotjosh <josue.abreu@gmail.com> * Update pkg/prom/instance/manager.go Co-authored-by: gotjosh <josue.abreu@gmail.com> * Update pkg/prom/instance/manager.go Co-authored-by: gotjosh <josue.abreu@gmail.com> * Update pkg/prom/instance/manager.go Co-authored-by: gotjosh <josue.abreu@gmail.com> Co-authored-by: gotjosh <josue.abreu@gmail.com>
mattdurham
pushed a commit
that referenced
this pull request
Nov 11, 2021
) * allow dynamic updating of instance scrape_configs and remote_write This commit introduces an Update method against the Instance that allows for applying new changes found on remote_write and scrape_configs sections for a particular instance. If any other field changes, Update will return ErrInvalidUpdate, which will cause the InstanceManager to do a normal "restart" of the instance. Future commits may loosen the restriction on what is able to be dynamically updated. Some code cleanup can be found as part of this commit: fakes that have been called mocks have been updated to use their proper name. * address review feedback that I gave myself * address review feedback that I didn't give myself
mattdurham
pushed a commit
that referenced
this pull request
Nov 11, 2021
* move InstanceManager to instance package This commit migrates InstanceManager to instance.BasicManager. This is to reduce duplication as we add more implementations of the instance.Manager interface (including removing ha.InstanceManager, which was a duplicate of that interface). While creating this commit, I noticed that there was a leaked locked mutex when DeleteConfig fails. That has been fixed with this commit. Related to the work for #141. Depends on #145. * Update pkg/prom/instance/manager.go Co-authored-by: gotjosh <josue.abreu@gmail.com> * Update pkg/prom/instance/manager.go Co-authored-by: gotjosh <josue.abreu@gmail.com> * Update pkg/prom/instance/manager.go Co-authored-by: gotjosh <josue.abreu@gmail.com> * Update pkg/prom/instance/manager.go Co-authored-by: gotjosh <josue.abreu@gmail.com> * Update pkg/prom/instance/manager.go Co-authored-by: gotjosh <josue.abreu@gmail.com> Co-authored-by: gotjosh <josue.abreu@gmail.com>
wildum
pushed a commit
that referenced
this pull request
Apr 9, 2024
github-actions
bot
added
the
frozen-due-to-age
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces an
Updatemethod against theInstancethat allows for applying new changes found onremote_writeandscrape_configssections. If any other field changes,Updatewill returnErrInvalidUpdate, which will cause theInstanceManagerto do the normal "restart" (i.e., shut down old instance, bring up new one with modified config).Future changes may loosen the restriction on what is able to be dynamically updated, but I suspect that
scrape_configsandremote_writeare going to be far more common than anything else.Some code cleanup can be found as part of this PR: fakes that have been called mocks have been updated to use their proper name.
Closes #140.