rule: fix reload signal not working #4442

jmichalek132 · 2021-07-13T20:20:21Z

Signed-off-by: Juraj Michalek juraj.michalek132@gmail.com

I added CHANGELOG entry for this change.
Change is not relevant to the end user.

Changes

Fixed a bug when sending sighup signal to thanos ruler didn't result in loading changes.

Verification

Started thanos locally using make quickstart.
Modified recording rule in rules.yml created by the quickstart option
Sent sighup to thanos ruler
Looked at thanos ruler ui to check if changes were successfully loaded (they were)
Also added e2e test for it

Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com>

GiedriusS · 2021-07-14T13:45:22Z

Thanks for the PR! 💪 Maybe we could add a test for this? Because this functionality was working, then we broke it, and now this fixes it again. Having a test would make sure that this works. What do you think? 🤔 The seems like a good fit for the e2e tests however you will probably have to improve the e2e framework for Cortex to add a KillSignal(signal int) error interface or something like that.

jmichalek132 · 2021-07-14T18:47:32Z

Yeah I was thinking adding the test would make sense too. Should I do it as part of this pr or could it be a follow up pr? Since making changes necessary in cortex might take some time.

I was also thinking exposing api to exec commands inside the containers started by the cortex e2e framework might make a bit more sense since it could be useful in other cases not just this one.

jmichalek132 · 2021-07-14T23:17:09Z

So it seems the e2e services already provides Exec which can be used to send the signal, now I just need to figure out how to check that the reload actually worked.

Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com>

jmichalek132 · 2021-07-15T13:34:34Z

I added the test for reloading using sighup. Currently I am checking that there 3 rule groups (there are 2 before the reload, one is added by the reload). Is the test okay like this?

jmichalek132 · 2021-07-15T14:13:48Z

Not sure why the unit tests failed but I don't think it's caused by my change. Could we re-run them?

GiedriusS

Just a few nits but overall LGTM 💪 I hope my suggestion is clear here, feel free to ask any questions 😄

test/e2e/rule_test.go

GiedriusS · 2021-07-15T14:12:43Z

test/e2e/rule_test.go

+	resp, err := http.DefaultClient.Do(req)
+	testutil.Ok(t, err)
+	testutil.Equals(t, 200, resp.StatusCode)
+	defer resp.Body.Close()


Let's just do: testutil.Ok(t, resp.Body.Close())

@jmichalek132 could you please simply do testutil.Ok(t, resp.Body.Close()) after ioutil.ReadAll()? It will be simpler + you'll get a failure if Close() fails

test/e2e/rule_test.go

Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com>

GiedriusS

Awesome work! Thanks ❤️ could you please do the last small change with regards to the defer resp.Body.Close()?

GiedriusS · 2021-07-19T06:41:44Z

test/e2e/rule_test.go

+	resp, err := http.DefaultClient.Do(req)
+	testutil.Ok(t, err)
+	testutil.Equals(t, 200, resp.StatusCode)
+	defer resp.Body.Close()


@jmichalek132 could you please simply do testutil.Ok(t, resp.Body.Close()) after ioutil.ReadAll()? It will be simpler + you'll get a failure if Close() fails

Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com>

test/e2e/rule_test.go

Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com>

* rule: fix reload signal not working Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com> * CHANGELOG: added PR Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com> * rule: added test for realoding using sighup signal Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com> * rule: adressed comments for test of realoding using sighup Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com> * ran make format Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com> * addressed comment for test of realoding using sighup Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com> * make linter happy Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com>

* rule: fix reload signal not working Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com> * CHANGELOG: added PR Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com> * rule: added test for realoding using sighup signal Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com> * rule: adressed comments for test of realoding using sighup Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com> * ran make format Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com> * addressed comment for test of realoding using sighup Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com> * make linter happy Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com> Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

jmichalek132 added 2 commits July 10, 2021 22:54

rule: fix reload signal not working

575cb09

Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com>

CHANGELOG: added PR

ddec899

Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com>

GiedriusS mentioned this pull request Jul 15, 2021

Thanos Rule fails to load all rules files after reload - numFiles is dropping #4445

Closed

rule: added test for realoding using sighup signal

541cacc

Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com>

GiedriusS reviewed Jul 15, 2021

View reviewed changes

jmichalek132 added 2 commits July 16, 2021 15:27

rule: adressed comments for test of realoding using sighup

5ac9691

Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com>

ran make format

3f4a4ad

Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com>

jmichalek132 requested a review from GiedriusS July 16, 2021 16:41

GiedriusS previously approved these changes Jul 19, 2021

View reviewed changes

addressed comment for test of realoding using sighup

f96b9bb

Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com>

jmichalek132 dismissed GiedriusS’s stale review via f96b9bb July 19, 2021 11:23

jmichalek132 requested a review from GiedriusS July 19, 2021 11:24

sonatype-lift bot reviewed Jul 19, 2021

View reviewed changes

test/e2e/rule_test.go Outdated Show resolved Hide resolved

make linter happy

c20d14a

Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com>

GiedriusS approved these changes Jul 19, 2021

View reviewed changes

GiedriusS merged commit e9ef82b into thanos-io:main Jul 19, 2021

GiedriusS mentioned this pull request Jul 19, 2021

[Thanos Rule] Rules Files not reloading after SIGHUP signal #4432

Closed

GiedriusS mentioned this pull request Jul 20, 2021

Release 0.22 #4457

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rule: fix reload signal not working #4442

rule: fix reload signal not working #4442

jmichalek132 commented Jul 13, 2021 •

edited

Loading

GiedriusS commented Jul 14, 2021

jmichalek132 commented Jul 14, 2021

jmichalek132 commented Jul 14, 2021 •

edited

Loading

jmichalek132 commented Jul 15, 2021

jmichalek132 commented Jul 15, 2021

GiedriusS left a comment

GiedriusS Jul 15, 2021

GiedriusS Jul 19, 2021

jmichalek132 Jul 19, 2021

GiedriusS left a comment

GiedriusS Jul 19, 2021

rule: fix reload signal not working #4442

rule: fix reload signal not working #4442

Conversation

jmichalek132 commented Jul 13, 2021 • edited Loading

Changes

Verification

GiedriusS commented Jul 14, 2021

jmichalek132 commented Jul 14, 2021

jmichalek132 commented Jul 14, 2021 • edited Loading

jmichalek132 commented Jul 15, 2021

jmichalek132 commented Jul 15, 2021

GiedriusS left a comment

Choose a reason for hiding this comment

GiedriusS Jul 15, 2021

Choose a reason for hiding this comment

GiedriusS Jul 19, 2021

Choose a reason for hiding this comment

jmichalek132 Jul 19, 2021

Choose a reason for hiding this comment

GiedriusS left a comment

Choose a reason for hiding this comment

GiedriusS Jul 19, 2021

Choose a reason for hiding this comment

jmichalek132 commented Jul 13, 2021 •

edited

Loading

jmichalek132 commented Jul 14, 2021 •

edited

Loading