Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Name or classify tasks spawned by the subscriber runtime #109

Open
seanmonstar opened this issue Sep 2, 2021 · 4 comments
Open

Name or classify tasks spawned by the subscriber runtime #109

seanmonstar opened this issue Sep 2, 2021 · 4 comments
Labels
C-subscriber Crate: console-subscriber. S-refactor Severity: refactor.

Comments

@seanmonstar
Copy link
Member

The subscriber starts its own runtime and server, and those tasks are instrumented and shown in the Console too. That is actually useful in some situations. But it also can be distracting. It would help if we could name or classify any tasks in that runtime to help a user viewing the Console, so they understand "oh, those aren't my tasks".

@seanmonstar seanmonstar added C-subscriber Crate: console-subscriber. S-refactor Severity: refactor. labels Sep 2, 2021
@hawkw
Copy link
Member

hawkw commented Sep 2, 2021

Since using the console already requires enabling tokio_unstable, we can probably just use task::Builder to add a name, anywhere we spawn a task.

In the future, it would be nice to be able to associate tasks with runtimes, and name runtimes as well. That way, we could say things like "hide all tasks belonging to the console runtime". But that would require new instrumentation in Tokio, so it's a bigger project.

hawkw added a commit that referenced this issue Sep 6, 2021
This branch updates the `console-subscriber` crate so that the
"aggregate" and "serve" tasks spawned by the console are spawned with
names. This allows the user to distinguish between tasks spawned by the
console subscriber and tasks spawned by other parts of the application.

Note that this does *not* include the tasks spawned by Tonic to handle
each accepted client connection. Those tasks are spawned inside of
Tonic, rather than by the console-subscriber crate, so we can't easily
add our own names to them. We could probably fix that by using Tonic's
lower level APIs to spawn our own per-connection tasks, and give them
names...but that seems like a follow-up PR.

This is a first pass on #109 (although it isn't a *complete* solution
due to the above issue with Tonic).
@hawkw
Copy link
Member

hawkw commented Sep 6, 2021

#117 added names to the console's aggregate and serve tasks. However, we can't easily name the per-connection tasks spawned by Tonic without switching to tonic's lower-level API and writing our own accept loop to spawn those tasks on our own. We'll do that in a subsequent branch (or consider changes upstream?)

hawkw added a commit that referenced this issue Sep 6, 2021
This branch updates the `console-subscriber` crate so that the
"aggregate" and "serve" tasks spawned by the console are spawned with
names. This allows the user to distinguish between tasks spawned by the
console subscriber and tasks spawned by other parts of the application.

Note that this does *not* include the tasks spawned by Tonic to handle
each accepted client connection. Those tasks are spawned inside of
Tonic, rather than by the console-subscriber crate, so we can't easily
add our own names to them. We could probably fix that by using Tonic's
lower level APIs to spawn our own per-connection tasks, and give them
names...but that seems like a follow-up PR.

This is a first pass on #109 (although it isn't a *complete* solution
due to the above issue with Tonic).

Screenshot: 
![image](https://user-images.githubusercontent.com/2796466/132248558-8417f2c8-317b-4d2a-85a9-c851dacba587.png)
@g2p
Copy link
Contributor

g2p commented Dec 20, 2021

Could the many tonic / hyper tasks be filtered out using spans and layers?
I don't know enough about tracing to know if logic like metadata.is_within_span is implementable.
Edit: something like a dynamic_filter_fn?
When debugging a hyper-based application tonic adds a lot of noise, ways to hide or at least distinguish would be welcome.

@hawkw
Copy link
Member

hawkw commented Dec 20, 2021

@g2p it's definitely possible to use dynamic_filter_fn to filter out those spans at the subscriber-level, but I think the best long-term solution is to add filtering to the console UI: #131

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-subscriber Crate: console-subscriber. S-refactor Severity: refactor.
Projects
None yet
Development

No branches or pull requests

3 participants