Discussion: ResourceManager defaults when active by default #9322
Assignees
Labels
kind/enhancement
A net-new feature or improvement to an existing feature
Comments
ajnavarro
added
the
kind/enhancement
|
@ajnavarro : I have spent some time looking at this and will type up some thoughts once I get out of meetings this morning. |
BigLep
added a commit
that referenced
this issue
Oct 18, 2022
The goal of this PR is to show an easier set of defaults for resource manager to reason about. This is an attempt to address #9322 The basic idea is: 1. Use these inputs: - maxMemory: can be set by user or default 1/8th the system memory - maxFD: can be set by user or default 1/2 the system limit - maxConns: can be set as the connection manager high water mark or defaults to infinity 2. Only set limits at the system and transient scope, and even there, mostly just focus on memory, FD, and inbound connections. Ingore outbout connections and stream limits. 3. Apply any limits that libp2p has for its protocols/services. This PR is not intended to be merged as is. It's not complete, undoubtedly has syntax errors, I haven't run tests, etc. It was done as a starting point to communicate specifically on how I think we can simplify the default story.
|
@ajnavarro : I was finding it easiest to convey my ideas in code. Here's a PR for discussion: #9351 Basically I imagine 3 options for users:
I'm also good if we want to simplify further. |
I think the cases where you're seeing limits == 0 is for cases where that value won't be consulted. For example, taking "libp2p.autonat"'s Conn limit of 0, that makes sense because that scope won't be consulted for connection limiting. |
20 tasks
|
We are getting these ideas actualized in #9338 |
ajnavarro
added a commit
that referenced
this issue
Nov 10, 2022
This PR adds several new functionalities to make easier the usage of ResourceManager: - Now resource manager logs when resources are exceeded are on ERROR instead of warning. - The resources exceeded error now shows what kind of limit was reached and the scope. - When there was no limit exceeded, we print a message for the user saying that limits are not exceeded anymore. - Added `swarm limit all` command to show all set limits with the same format as `swarm stats all` - Added `min-used-limit-perc` option to `swarm stats all` to only show stats that are above a specific percentage - Simplify a lot default values. - **Enable ResourceManager by default.** Output example: ``` 2022-11-09T10:51:40.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:59 Consider inspecting logs and raising the resource manager limits. Documentation: https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmresourcemgr 2022-11-09T10:51:50.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:55 Resource limits were exceeded 483095 times with error "transient: cannot reserve inbound stream: resource limit exceeded". 2022-11-09T10:51:50.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:59 Consider inspecting logs and raising the resource manager limits. Documentation: https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmresourcemgr 2022-11-09T10:52:00.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:55 Resource limits were exceeded 455294 times with error "transient: cannot reserve inbound stream: resource limit exceeded". 2022-11-09T10:52:00.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:59 Consider inspecting logs and raising the resource manager limits. Documentation: https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmresourcemgr 2022-11-09T10:52:10.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:55 Resource limits were exceeded 471384 times with error "transient: cannot reserve inbound stream: resource limit exceeded". 2022-11-09T10:52:10.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:59 Consider inspecting logs and raising the resource manager limits. Documentation: https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmresourcemgr 2022-11-09T10:52:20.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:55 Resource limits were exceeded 8 times with error "peer:12D3KooWKqcaBtcmZKLKCCoDPBuA6AXGJMNrLQUPPMsA5Q6D1eG6: cannot reserve inbound stream: resource limit exceeded". 2022-11-09T10:52:20.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:55 Resource limits were exceeded 192 times with error "peer:12D3KooWPjetWPGQUih9LZTGHdyAM9fKaXtUxDyBhA93E3JAWCXj: cannot reserve inbound stream: resource limit exceeded". 2022-11-09T10:52:20.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:55 Resource limits were exceeded 469746 times with error "transient: cannot reserve inbound stream: resource limit exceeded". 2022-11-09T10:52:20.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:59 Consider inspecting logs and raising the resource manager limits. Documentation: https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmresourcemgr 2022-11-09T10:52:30.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:55 Resource limits were exceeded 484137 times with error "transient: cannot reserve inbound stream: resource limit exceeded". 2022-11-09T10:52:30.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:55 Resource limits were exceeded 29 times with error "peer:12D3KooWPjetWPGQUih9LZTGHdyAM9fKaXtUxDyBhA93E3JAWCXj: cannot reserve inbound stream: resource limit exceeded". 2022-11-09T10:52:30.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:59 Consider inspecting logs and raising the resource manager limits. Documentation: https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmresourcemgr 2022-11-09T10:52:40.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:55 Resource limits were exceeded 468843 times with error "transient: cannot reserve inbound stream: resource limit exceeded". 2022-11-09T10:52:40.566+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:59 Consider inspecting logs and raising the resource manager limits. Documentation: https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmresourcemgr 2022-11-09T10:52:50.566+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:55 Resource limits were exceeded 366638 times with error "transient: cannot reserve inbound stream: resource limit exceeded". 2022-11-09T10:52:50.566+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:59 Consider inspecting logs and raising the resource manager limits. Documentation: https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmresourcemgr 2022-11-09T10:53:00.566+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:55 Resource limits were exceeded 405526 times with error "transient: cannot reserve inbound stream: resource limit exceeded". 2022-11-09T10:53:00.566+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:55 Resource limits were exceeded 107 times with error "peer:12D3KooWQZQCwevTDGhkE9iGYk5sBzWRDUSX68oyrcfM9tXyrs2Q: cannot reserve inbound stream: resource limit exceeded". 2022-11-09T10:53:00.566+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:59 Consider inspecting logs and raising the resource manager limits. Documentation: https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmresourcemgr 2022-11-09T10:53:10.566+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:55 Resource limits were exceeded 336923 times with error "transient: cannot reserve inbound stream: resource limit exceeded". 2022-11-09T10:53:10.566+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:59 Consider inspecting logs and raising the resource manager limits. Documentation: https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmresourcemgr 2022-11-09T10:53:20.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:55 Resource limits were exceeded 71 times with error "transient: cannot reserve inbound stream: resource limit exceeded". 2022-11-09T10:53:20.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:59 Consider inspecting logs and raising the resource manager limits. Documentation: https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmresourcemgr 2022-11-09T10:53:30.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:64 Resrouce limits are no longer being exceeded. ``` ## Validation tests - Accelerated DHT client runs with no errors when ResourceManager is active. No problems were observed. - Running an attack with 200 connections and 1M streams using yamux protocol. Node was usable during the attack. With ResourceManager deactivated, the node was killed by the OS because of the amount of memory consumed. - Actions done when the attack was active: - Add files - Force a reprovide - Use the gateway to resolve an IPNS address. It closes #9001 It closes #9351 It closes #9322
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This is the default resource manager configuration on a machine with 32Gb of memory, and 65536 file descriptors:
Default ResourceManager config
{ "System": { "Streams": 65536, "StreamsInbound": 16384, "StreamsOutbound": 65536, "Conns": 4096, "ConnsInbound": 2048, "ConnsOutbound": 2048, "FD": 4096, "Memory": 4331667456 }, "Transient": { "Streams": 1256, "StreamsInbound": 628, "StreamsOutbound": 1256, "Conns": 189, "ConnsInbound": 94, "ConnsOutbound": 189, "FD": 1024, "Memory": 558235648 }, "AllowlistedSystem": { "Streams": 10054, "StreamsInbound": 5027, "StreamsOutbound": 10054, "Conns": 628, "ConnsInbound": 314, "ConnsOutbound": 628, "FD": 4096, "Memory": 4331667456 }, "AllowlistedTransient": { "Streams": 1256, "StreamsInbound": 628, "StreamsOutbound": 1256, "Conns": 189, "ConnsInbound": 94, "ConnsOutbound": 189, "FD": 1024, "Memory": 558235648 }, "ServiceDefault": { "Streams": 32768, "StreamsInbound": 8192, "StreamsOutbound": 32768, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 591790080 }, "Service": { "libp2p.autonat": { "Streams": 79, "StreamsInbound": 79, "StreamsOutbound": 79, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 12392448 }, "libp2p.holepunch": { "Streams": 126, "StreamsInbound": 63, "StreamsOutbound": 63, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 20590592 }, "libp2p.identify": { "Streams": 628, "StreamsInbound": 314, "StreamsOutbound": 314, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 20590592 }, "libp2p.ping": { "Streams": 314, "StreamsInbound": 314, "StreamsOutbound": 314, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 20590592 }, "libp2p.relay/v1": { "Streams": 1256, "StreamsInbound": 1256, "StreamsOutbound": 1256, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 82362368 }, "libp2p.relay/v2": { "Streams": 1256, "StreamsInbound": 1256, "StreamsOutbound": 1256, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 82362368 } }, "ServicePeerDefault": { "Streams": 287, "StreamsInbound": 143, "StreamsOutbound": 287, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 33173504 }, "ServicePeer": { "libp2p.autonat": { "Streams": 2, "StreamsInbound": 2, "StreamsOutbound": 2, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 1048576 }, "libp2p.holepunch": { "Streams": 2, "StreamsInbound": 2, "StreamsOutbound": 2, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 1048576 }, "libp2p.identify": { "Streams": 32, "StreamsInbound": 16, "StreamsOutbound": 16, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 1048576 }, "libp2p.ping": { "Streams": 4, "StreamsInbound": 2, "StreamsOutbound": 3, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 8590458880 }, "libp2p.relay/v1": { "Streams": 64, "StreamsInbound": 64, "StreamsOutbound": 64, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 1048576 }, "libp2p.relay/v2": { "Streams": 64, "StreamsInbound": 64, "StreamsOutbound": 64, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 1048576 } }, "ProtocolDefault": { "Streams": 32768, "StreamsInbound": 8192, "StreamsOutbound": 32768, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 739356672 }, "Protocol": { "/ipfs/id/1.0.0": { "Streams": 628, "StreamsInbound": 314, "StreamsOutbound": 314, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 20590592 }, "/ipfs/id/push/1.0.0": { "Streams": 628, "StreamsInbound": 314, "StreamsOutbound": 314, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 20590592 }, "/ipfs/ping/1.0.0": { "Streams": 314, "StreamsInbound": 314, "StreamsOutbound": 314, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 20590592 }, "/libp2p/autonat/1.0.0": { "Streams": 79, "StreamsInbound": 79, "StreamsOutbound": 79, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 12392448 }, "/libp2p/circuit/relay/0.1.0": { "Streams": 3141, "StreamsInbound": 3141, "StreamsOutbound": 3141, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 82362368 }, "/libp2p/circuit/relay/0.2.0/hop": { "Streams": 3141, "StreamsInbound": 3141, "StreamsOutbound": 3141, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 82362368 }, "/libp2p/circuit/relay/0.2.0/stop": { "Streams": 3141, "StreamsInbound": 3141, "StreamsOutbound": 3141, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 82362368 }, "/libp2p/dcutr": { "Streams": 126, "StreamsInbound": 63, "StreamsOutbound": 63, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 20590592 }, "/p2p/id/delta/1.0.0": { "Streams": 628, "StreamsInbound": 314, "StreamsOutbound": 314, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 20590592 } }, "ProtocolPeerDefault": { "Streams": 318, "StreamsInbound": 79, "StreamsOutbound": 159, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 16777231 }, "ProtocolPeer": { "/ipfs/id/1.0.0": { "Streams": 32, "StreamsInbound": 16, "StreamsOutbound": 16, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 8590458880 }, "/ipfs/id/push/1.0.0": { "Streams": 32, "StreamsInbound": 16, "StreamsOutbound": 16, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 8590458880 }, "/ipfs/ping/1.0.0": { "Streams": 4, "StreamsInbound": 2, "StreamsOutbound": 3, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 8590458880 }, "/libp2p/autonat/1.0.0": { "Streams": 2, "StreamsInbound": 2, "StreamsOutbound": 2, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 1048576 }, "/libp2p/circuit/relay/0.1.0": { "Streams": 128, "StreamsInbound": 128, "StreamsOutbound": 128, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 33554432 }, "/libp2p/circuit/relay/0.2.0/hop": { "Streams": 128, "StreamsInbound": 128, "StreamsOutbound": 128, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 33554432 }, "/libp2p/circuit/relay/0.2.0/stop": { "Streams": 128, "StreamsInbound": 128, "StreamsOutbound": 128, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 33554432 }, "/libp2p/dcutr": { "Streams": 2, "StreamsInbound": 2, "StreamsOutbound": 2, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 1048576 }, "/p2p/id/delta/1.0.0": { "Streams": 32, "StreamsInbound": 16, "StreamsOutbound": 16, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 8590458880 } }, "PeerDefault": { "Streams": 1512, "StreamsInbound": 756, "StreamsOutbound": 1512, "Conns": 8, "ConnsInbound": 4, "ConnsOutbound": 8, "FD": 64, "Memory": 591790080 }, "Conn": { "Streams": 0, "StreamsInbound": 0, "StreamsOutbound": 0, "Conns": 1, "ConnsInbound": 1, "ConnsOutbound": 1, "FD": 1, "Memory": 33554432 }, "Stream": { "Streams": 1, "StreamsInbound": 1, "StreamsOutbound": 1, "Conns": 0, "ConnsInbound": 0, "ConnsOutbound": 0, "FD": 0, "Memory": 16777216 } }Right now, auto-scaling functionality is getting half of the file descriptors and 1/8 of the memory.
From Kubo side, we are modifying several specific parameters:
SystemBaseLimit.ConnsOutbound: we are setting it to 65536 if the default value is lower. It is done to allow the accelerated DHT to load its routing table.SystemBaseLimit.FDis set to 4096 if the default value is lower than that.ConnMgr.Typetype isbasicwe set other extra params.We use
ConnMgr.HighWater(by default 900) as a base to configure the following params IFSystem.ConnsInboundis smaller than 2*HighWaterAll the following commands are set using a function that converts the provided
HighWaterwith a multiplier that basically counts the number of needed bits to represent that number and adds these number of bits to 1. For example, 0 will need 0 bits to represent it, so the output is 1. 10 needs 4 bits, so the output will be 1 << 4 = 16, and so on (do not ask me why we are doing this.).System.ConnsInbound: 2*HighWaterSystem.ConnsOutbound: 2*HighWaterSystem.Conns: 4*HighWaterSystem.StreamsInbound: 16*HighWaterSystem.StreamsOutbound: 64*HighWaterSystem.Streams: 64*HighWaterSystem.FD: 2*HighWaterServiceDefault.StreamsInbound: 8*HighWaterServiceDefault.StreamsOutbound: 32*HighWaterServiceDefault.Streams: 32*HighWaterProtocolDefault.StreamsInbound: 8*HighWaterProtocolDefault.StreamsOutbound: 32*HighWaterProtocolDefault.Streams: 32*HighWaterThese values will be the default ones when we activate ResourceManager by default. Right now, ResourceManager is not managing resources but mostly limiting them, throwing errors internally when limits are reached.
Also, as you can see on the default configuration, we have some values set to 0. It is still not clear to me if that value means no limit or limit == 0. Some conversation about that here: https://filecoinproject.slack.com/archives/C03FFEVK30F/p1664184608359269
Related issue: #8761
CC: @guseggert , @Jorropo , @lidel , @BigLep , @galargh WDYT about these defaults? do they look good for you? Thx.
The text was updated successfully, but these errors were encountered: