fix(console): fix the issue of continuous reconnection caused by decode failure

[ethercflow]

2023-12-12T08:46:56.332653Z WARN tokio_console::conn: error from stream status=status: OutOfRange, message: "Error, message length too large: found 5913832 bytes, the limit is: 4194304 bytes", details: [], metadata: MetadataMap { headers: {} }

By default, both of the tonic's max_decoding_message_size is 4M, when there are too many async tasks, the decoding will over 4M, causing tokio-console decoding to fail.

When it fails, the tokio-console interface will continuously display Connected/Reconnect, as it may be a bit strange to make the decoding maximum value a command line parameter, so make it 16M as a const value.

[grahamking]

I hit the same error and your patch fixes it. I can connect to my server shortly after it starts, but if I wait a few minutes before connecting I start seeing that log message and tokio-console won't work.

Thanks!

UPDATE: The fix only buys some time but doesn't solve the problem. I eventually got to:

2023-12-12T22:32:31.452815Z WARN tokio_console::conn: error from stream status=status: OutOfRange, message: "Error, message length too large: found 18007258 bytes, the limit is: 16777216 bytes", details: [], metadata: MetadataMap { headers: {} }

Not sure if it's my app or tokio-console. My guess is it's console-subscriber.

[ethercflow]

I hit the same error and your patch fixes it. I can connect to my server shortly after it starts, but if I wait a few minutes before connecting I start seeing that log message and tokio-console won't work.

Thanks!

UPDATE: The fix only buys some time but doesn't solve the problem. I eventually got to:

2023-12-12T22:32:31.452815Z WARN tokio_console::conn: error from stream status=status: OutOfRange, message: "Error, message length too large: found 18007258 bytes, the limit is: 16777216 bytes", details: [], metadata: MetadataMap { headers: {} }

Not sure if it's my app or tokio-console. My guess is it's console-subscriber.

Thank you for the information. Maybe we also need to limit the size of encoding. I will do further research to confirm.

[ethercflow]

@grahamking From https://github.com/hyperium/tonic/blob/v0.10.2/tonic/src/lib.rs#L62 it says:

//! # Code generated client/server configuration
//!
//! ## Max Message Size
//!
//! Currently, both servers and clients can be configured to set the max message encoding and
//! decoding size. This will ensure that an incoming gRPC message will not exahust the systems
//! memory. By default, the decoding message limit is 4MB and the encoding limit is usize::MAX.

So I think we need to call max_encoding__message_size to limit memory usage in server side. I'm curious, how many async tasks do you have there that generate so much data?

[grahamking]

I don't have very many tasks, maybe 20 per second at most, they are short lived.

I think the issue is that when a client first connects it gets an Update object which contains all data (tasks, resources, async_ops) for the past retention, which defaults to 1 hour. The encoded length of that object can quite easily get above 4MB.
This object: https://github.com/tokio-rs/console/blob/main/console-subscriber/src/aggregator/mod.rs#L283-L291

Reducing retention to 1 minute prevents the problem, but then I get a different issue where nothing updates in the tokio-console display. Still trying to figure that one out.

[ethercflow]

I don't have very many tasks, maybe 20 per second at most, they are short lived.

I think the issue is that when a client first connects it gets an Update object which contains all data (tasks, resources, async_ops) for the past retention, which defaults to 1 hour. The encoded length of that object can quite easily get above 4MB. This object: https://github.com/tokio-rs/console/blob/main/console-subscriber/src/aggregator/mod.rs#L283-L291

Reducing retention to 1 minute prevents the problem, but then I get a different issue where nothing updates in the tokio-console display. Still trying to figure that one out.

Oh I see, indeed, keeping 1h by default consumes a lot of memory.

[grahamking]

I think I hit two overlapping problems:

• If I don't set retention to a short enough duration the tasks/resources/async_ops data will grow beyond the message size.
• Even with retention, if I don't connect often enough the poll_ops (which aren't affected by retention) will grow too large for a message. This fixes it for me: fix(subscriber): Don't save poll_ops if no-one is receiving them #501