Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CORE-96]: admin/server fix set_log_level handling of overlapping expirations #18397

Conversation

andijcr
Copy link
Contributor

@andijcr andijcr commented May 10, 2024

This PR fixes this two behaviors

overlapping expiration of different loggers

before

one logger expiration could shadow another logger expiration

log-level set kafka -l trace -e 30
log-level set raft -l trace -e 30
.. 30+ secs later
log-level get kafka -> info
log-level get raft -> trace <- not being reset

with this pr

both loggers are reset correctly

log-level set kafka -l trace -e 30
log-level set raft -l trace -e 30
.. 30+ secs later
log-level get kafka -> info
log-level get raft -> info <- correctly reset

expire-never request not being honored, if overlapping a previous request

a logger expiration could influence a subsequent request to set a level without expiration
note: this case in practice is probably never triggered, since the only level that shows anomalous behavior is error and warn, and they are not levels that are normally used

before

log-level set raft -l trace -e 30
log-level set raft -l error -e 0 <- 0 is for expire-never
... 30+ secs later
log-level get raft -> info <- reset to a default level

with this pr

log-level set raft -l trace -e 30
log-level set raft -l error -e 0 <- 0 is for expire-never
... 30+ secs later
log-level get raft -> error <- keeps its value

Fixes CORE-96

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v24.1.x
  • v23.3.x
  • v23.2.x

Release Notes

Bug Fixes

  • concurrent requests of set_log_level + expiration now work as expected

andijcr added 2 commits May 10, 2024 17:21
_log_level_resets is a map<str, level_reset>, we want to extract the
next expiring level_reset to rearm the _log_level_timer.

previously we would extract the min_element based on the key (the logger
name)

so this sequence of actions
set log-level kafka trace 10
set log-level raft trace 10

only kafka would be expired, because once expired the next run would
pick again kafka, see that there is no expiration, and not rearm the
timer.
andijcr added 3 commits May 10, 2024 18:33
the test shows that this sequence of actions

set log-level admin-api-server trace 10
set log-level raft trace 10

works and both logger are reset after the expiry period
previously, this sequence of actions

set log-level raft trace 10
set log-level raft error 0

would result in raft logging at error level for only 10 seconds, despite
the legit request to expire never.

this is because in the code, the second request would not correctly
reset the expire field if a previous name-level_reset was already in the
map used for this operation
test that
set log-level admin_api_server trace 10
set log-level admin_api_server error 0

never resets the logger to the info level
@andijcr andijcr force-pushed the fix/trace_logging_stacked_deactivation branch from 48db97d to 84748a8 Compare May 10, 2024 16:33
@andijcr andijcr requested review from a team, pgellert and oleiman and removed request for a team May 10, 2024 16:34
Copy link
Member

@oleiman oleiman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for fixing those 😅

@andijcr
Copy link
Contributor Author

andijcr commented May 13, 2024

failure is #13275 (just closed, at the time of writing )

@michael-redpanda michael-redpanda merged commit e10e6f2 into redpanda-data:dev May 13, 2024
15 of 18 checks passed
@vbotbuildovich
Copy link
Collaborator

/backport v24.1.x

@vbotbuildovich
Copy link
Collaborator

/backport v23.3.x

@vbotbuildovich
Copy link
Collaborator

/backport v23.2.x

@vbotbuildovich
Copy link
Collaborator

Failed to create a backport PR to v23.2.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-18397-v23.2.x-967 remotes/upstream/v23.2.x
git cherry-pick -x bf6bc91ae98162e9504cb33fbffde62f5045211c 0717cd1a0c9f352ad8a66d6cdf10ae58f36147a1 6593dbec194446702799c3223caeb688000019cf d781fb13e6051bf3b01821b9faac4687fa69f1c6 84748a8dcb4931258c5b9ef0cb3a32b9f2b9202b

Workflow run logs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants