Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: allow updating export options at runtime #974

Merged
merged 1 commit into from
Aug 2, 2024

Conversation

TheSpiritXIII
Copy link
Member

@TheSpiritXIII TheSpiritXIII commented May 16, 2024

Pre-req: #1001

  1. ApplyConfig now takes ExporterOpts, which allows runtime updates.
  2. exporter.New now takes a context and Run no longer does, which means the Exporter uses the same context throughout its lifetime.

These changes allow us to update the metric client at runtime. When a metric client is updated, it may take:

  • 50ms worse case, when not enough samples are being exported to fill a batch.
  • <50ms best case, when many samples (>200) are being written

@TheSpiritXIII TheSpiritXIII force-pushed the TheSpiritXIII/export-runtime branch 8 times, most recently from 17fc0fd to a54ec88 Compare June 7, 2024 18:51
@TheSpiritXIII TheSpiritXIII marked this pull request as ready for review June 11, 2024 17:49
@TheSpiritXIII TheSpiritXIII changed the title [DRAFT]: feat: allow updating export options at runtime feat: allow updating export options at runtime Jun 11, 2024
@TheSpiritXIII TheSpiritXIII force-pushed the TheSpiritXIII/export-runtime branch 4 times, most recently from ffd0c0c to c09542e Compare June 18, 2024 20:54
Copy link
Collaborator

@pintohutch pintohutch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

Partial review - will finish up later this week.

pkg/export/export.go Show resolved Hide resolved
pkg/export/export.go Outdated Show resolved Hide resolved
pkg/export/export.go Outdated Show resolved Hide resolved
pkg/export/export.go Outdated Show resolved Hide resolved
pkg/export/export.go Show resolved Hide resolved
pkg/export/export.go Show resolved Hide resolved
pkg/export/export.go Show resolved Hide resolved
Copy link
Collaborator

@pintohutch pintohutch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apologies for the delay, finally finished review.

Looks like this is a combination change that depends on #1001. Is there any way this can be separated out? It's difficult to review the combined changes at once.

pkg/export/export.go Show resolved Hide resolved
pkg/export/export.go Outdated Show resolved Hide resolved
pkg/export/export.go Show resolved Hide resolved
pkg/export/export.go Show resolved Hide resolved
pkg/export/export.go Outdated Show resolved Hide resolved
pkg/export/setup/setup.go Show resolved Hide resolved
pkg/export/setup/setup.go Outdated Show resolved Hide resolved
pkg/export/setup/setup.go Outdated Show resolved Hide resolved
pkg/export/setup/setup_test.go Outdated Show resolved Hide resolved
cmd/rule-evaluator/main.go Outdated Show resolved Hide resolved
@TheSpiritXIII
Copy link
Member Author

Is there any way this can be separated out?

It already is -- go to the "Files changed" tab and on the top right directly underneath the tabs, there's a drop-down that says "All commits" where you can select which commit to review. Or do you mean you want this change without being rebased on top of the other change?

@pintohutch
Copy link
Collaborator

Ah ok - it wasn't clear to me that I should just review that commit.

If that's the case, I think most of my review comments outside of the rule-evaluator ones still apply - thanks for clarifying :)

@TheSpiritXIII TheSpiritXIII force-pushed the TheSpiritXIII/export-runtime branch 3 times, most recently from 1730089 to 7da8e19 Compare July 3, 2024 16:46
@TheSpiritXIII TheSpiritXIII changed the base branch from main to TheSpiritXIII/refactor-flags July 3, 2024 17:09
Base automatically changed from TheSpiritXIII/refactor-flags to main July 10, 2024 19:57
@TheSpiritXIII TheSpiritXIII force-pushed the TheSpiritXIII/export-runtime branch 4 times, most recently from 0978dc1 to 42954eb Compare July 17, 2024 19:23
cmd/rule-evaluator/main.go Show resolved Hide resolved
pkg/export/export.go Outdated Show resolved Hide resolved
pkg/export/export.go Show resolved Hide resolved
pkg/export/export.go Show resolved Hide resolved
pkg/export/storage.go Outdated Show resolved Hide resolved
pkg/export/export.go Show resolved Hide resolved
pkg/export/export.go Outdated Show resolved Hide resolved
pkg/export/export.go Show resolved Hide resolved
pkg/export/export.go Show resolved Hide resolved
Copy link
Member Author

@TheSpiritXIII TheSpiritXIII left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pintohutch thanks! I made some changes you requested. Please take another look!

pkg/export/storage.go Outdated Show resolved Hide resolved
cmd/rule-evaluator/main.go Show resolved Hide resolved
pkg/export/export.go Show resolved Hide resolved
pkg/export/export.go Show resolved Hide resolved
pkg/export/export.go Show resolved Hide resolved
pkg/export/export.go Outdated Show resolved Hide resolved
Copy link
Collaborator

@pintohutch pintohutch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome - I think it mostly LGTM, though with a change of this scope I'd love some co-review love from @bwplotka if possible.

A few comments and questions. Once those are resolved, I think it's good to :shipit:

pkg/export/export.go Show resolved Hide resolved
pkg/export/export.go Show resolved Hide resolved
pkg/export/export_test.go Show resolved Hide resolved
@TheSpiritXIII TheSpiritXIII force-pushed the TheSpiritXIII/export-runtime branch 2 times, most recently from 63a60d8 to 0755bbf Compare July 29, 2024 17:45
Copy link
Collaborator

@pintohutch pintohutch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

pkg/export/export.go Outdated Show resolved Hide resolved
pkg/export/export.go Outdated Show resolved Hide resolved
pkg/export/export.go Outdated Show resolved Hide resolved
Copy link
Collaborator

@bwplotka bwplotka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Generally LGTM, it's nice as it enables collector improvement of no restart for exporter config / label setting.

I would focus on killing testing code in prod code though, it's getting bit out of control, but I will understand if @TheSpiritXIII don't have time for that here. Plus I would NOT modify options (or config) in apply method, feels it can cause surprised going forward.

Copy link
Collaborator

@bwplotka bwplotka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for great work on this - it wasn't easy. What I like about this, is that it actually is more consistent with Prometheus OSS, PRW handling is hot-configurable 👍🏽

Thanks for addressing all tricky comments and reducing testing code in prod code 💪🏽

Great effort on tests too!

Comment on lines +343 to +344
_, err := google.FindDefaultCredentials(ctx)
if err != nil {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
_, err := google.FindDefaultCredentials(ctx)
if err != nil {
if _, err := google.FindDefaultCredentials(ctx); err != nil {

Copy link
Collaborator

@bwplotka bwplotka Aug 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small nit so merging PR without this

pkg/export/setup/setup.go Show resolved Hide resolved
@@ -74,7 +74,9 @@ func (l *localExportWithGCM) Ref() string { return "export-pkg-with-gcm" }
func (l *localExportWithGCM) start(t testing.TB, _ e2e.Environment) (v1.API, map[string]string) {
t.Helper()

ctx := context.Background()
ctx, cancel := context.WithCancel(signals.SetupSignalHandler())
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

woooo didn't know this, so good

@bwplotka bwplotka merged commit cfce8f9 into main Aug 2, 2024
27 checks passed
@TheSpiritXIII TheSpiritXIII deleted the TheSpiritXIII/export-runtime branch August 2, 2024 12:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants