command/operator_debug: add pprof interval #11938
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tgross
requested changes
Jan 26, 2022
There was a problem hiding this comment.
Hi @danishprakash! Overall this is looking pretty good. I've left some comments, but I'm also going to rope in @davemay99 for a review as he's kind of the internal "owner" of this feature.
danishprakash
force-pushed
the
operator-debug-pprof-interval
branch
from
27b5d0b
to
b5840d7
Compare
danishprakash
force-pushed
the
operator-debug-pprof-interval
branch
from
b5840d7
to
f2b7865
Compare
danishprakash
force-pushed
the
operator-debug-pprof-interval
branch
from
f2b7865
to
107f477
Compare
danishprakash
force-pushed
the
operator-debug-pprof-interval
branch
from
107f477
to
e36f1ca
Compare
danishprakash
force-pushed
the
operator-debug-pprof-interval
branch
from
e36f1ca
to
ef538ab
Compare
tgross
requested changes
Mar 7, 2022
|
|
|
Hey @danishprakash Just checking in to see if you're still interested in working on this. We think this'd be a great change and would love to see this get over the hump 😄 |
|
Hey @Amier3, I missed following up on this, I've pushed some updates, will try to close this soon :) Thanks for the heads up. |
Signed-off-by: danishprakash <grafitykoncept@gmail.com>
Signed-off-by: danishprakash <grafitykoncept@gmail.com>
Signed-off-by: danishprakash <grafitykoncept@gmail.com>
tgross
force-pushed
the
operator-debug-pprof-interval
branch
from
a4a0da4
to
215c80a
Compare
tgross
force-pushed
the
operator-debug-pprof-interval
branch
from
215c80a
to
c1f41c2
Compare
tgross
force-pushed
the
operator-debug-pprof-interval
branch
from
c1f41c2
to
e7c0317
Compare
tgross
force-pushed
the
operator-debug-pprof-interval
branch
from
e7c0317
to
23f26fb
Compare
tgross
force-pushed
the
operator-debug-pprof-interval
branch
from
23f26fb
to
69a473f
Compare
danishprakash
commented
Apr 5, 2022
tgross
added a commit
that referenced
this pull request
Apr 7, 2022
We introduced a `pprof-interval` argument to `operator debug` in #11938, and unfortunately this has resulted in a lot of test flakes. The actual command in use is mostly fine (although I've fixed some quirks here), so what's really happened is that the change has revealed some existing issues in the tests. Summary of changes: * Make first pprof collection synchronous to preserve the existing behavior for the common case where the pprof interval matches the duration. * Clamp `operator debug` pprof timing to that of the command. The `pprof-duration` should be no more than `duration` and the `pprof-interval` should be no more than `pprof-duration`. Clamp the values rather than throwing errors, which could change the commands that existing users might already have in debugging scripts * Testing: remove test parallelism The `operator debug` tests that stand up servers can't be run in parallel, because we don't have a way of canceling the API calls for pprof. The agent will still be running the last pprof when we exit, and that breaks the next test that talks to that same agent. (Because you can only run one pprof at a time on any process!) We could split off each subtest into its own server, but this test suite is already very slow. In future work we should fix this "for real" by making the API call cancelable. * Testing: assert against unexpected errors in `operator debug` tests. If we assert there are no unexpected error outputs, it's easier for the developer to debug when something is going wrong with the tests because the error output will be presented as a failing test, rather than just a failing exit code check. Or worse, no failing exit code check! This also forces us to be explicit about which tests will return 0 exit codes but still emit (presumably ignorable) error outputs. Additional minor bug fixes (mostly in tests) and test refactorings: * Fix text alignment on pprof Duration in `operator debug` output * Remove "done" channel from `operator debug` event stream test. The goroutine we're blocking for here already tells us it's done by sending a value, so block on that instead of an extraneous channel * Event stream test timer should start at current time, not zero * Remove noise from `operator debug` test log output. The `t.Logf` calls already are picked out from the rest of the test output by being prefixed with the filename. * Remove explicit pprof args so we use the defaults clamped from duration/interval
|
I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #11514