Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase in error rate after upgrade to v0.9.36 #748

Closed
BulatSaif opened this issue Dec 21, 2022 · 6 comments · Fixed by #1313 · May be fixed by paritytech/polkadot#7211
Closed

Increase in error rate after upgrade to v0.9.36 #748

BulatSaif opened this issue Dec 21, 2022 · 6 comments · Fixed by #1313 · May be fixed by paritytech/polkadot#7211

Comments

@BulatSaif
Copy link
Contributor

On 2022-12-20 16:50:21 UTC we updated Polkadot RPC nodes v0.9.33 -> V0.9.36, and after that, we see more errors in logs than usual.
image

Error:

ERROR tokio-runtime-worker jsonrpsee_server::transport::ws: WS transport error: i/o error: Transport endpoint is not connected (os error 107); terminate connection:

As far as I understand, the logs are due to broken WS connections.

Is the new release breaking the WS or did Polkadot enable more logs?

@bkchr
Copy link
Member

bkchr commented Dec 22, 2022

@niklasad1 do we really need to log these errors? Probably using debug logging should be enough?

@niklasad1
Copy link
Member

niklasad1 commented Dec 24, 2022

This is really a non-clean shutdown or if the socket if the websocket send failed but sure I think we can downgrade these to debug especially this is just noise for polkadot.

@bLd75
Copy link

bLd75 commented Mar 7, 2023

We also noticed another error on HTTP endpoints since upgrade, but in a much bigger proportion than on WS endpoints.
The message seems to be a warning but should we be concerned about requests dropped or is it just node internal warning?
https://substrate.stackexchange.com/questions/7485/parachain-rpc-errors-since-polkadot-0-9-36-client-upgrade

@niklasad1
Copy link
Member

@bLd75 That's something completely different and I tried to reply to your stackexchange question.

@Sophia-Gold Sophia-Gold transferred this issue from paritytech/polkadot Aug 24, 2023
@BulatSaif
Copy link
Contributor Author

related: paritytech/jsonrpsee#871

@niklasad1
Copy link
Member

@BulatSaif This will be fixed by upgrading jsonrpsee and the PR is up for review now.

claravanstaden added a commit to Snowfork/polkadot-sdk that referenced this issue Dec 8, 2023
* Upgrades to lodestar 1.3.0.

* Revert unrelated changes.

* Updates lodestar version in hack-ethereum.

* Updates lodestar minimal config file path.

Co-authored-by: claravanstaden <Cats 4 life!>
github-merge-queue bot pushed a commit that referenced this issue Jan 23, 2024
This is a rather big change in jsonrpsee, the major things in this bump
are:
- Server backpressure (the subscription impls are modified to deal with
that)
- Allow custom error types / return types (remove jsonrpsee::core::Error
and jsonrpee::core::CallError)
- Bug fixes (graceful shutdown in particular not used by substrate
anyway)
   - Less dependencies for the clients in particular
   - Return type requires Clone in method call responses
   - Moved to tokio channels
   - Async subscription API (not used in this PR)

Major changes in this PR:
- The subscriptions are now bounded and if subscription can't keep up
with the server it is dropped
- CLI: add parameter to configure the jsonrpc server bounded message
buffer (default is 64)
- Add our own subscription helper to deal with the unbounded streams in
substrate

The most important things in this PR to review is the added helpers
functions in `substrate/client/rpc/src/utils.rs` and the rest is pretty
much chore.

Regarding the "bounded buffer limit" it may cause the server to handle
the JSON-RPC calls
slower than before.

The message size limit is bounded by "--rpc-response-size" thus "by
default 10MB * 64 = 640MB"
but the subscription message size is not covered by this limit and could
be capped as well.

Hopefully the last release prior to 1.0, sorry in advance for a big PR

Previous attempt: paritytech/substrate#13992

Resolves #748, resolves
#627
franciscoaguirre pushed a commit to paritytech/xcm that referenced this issue Jan 25, 2024
This is a rather big change in jsonrpsee, the major things in this bump
are:
- Server backpressure (the subscription impls are modified to deal with
that)
- Allow custom error types / return types (remove jsonrpsee::core::Error
and jsonrpee::core::CallError)
- Bug fixes (graceful shutdown in particular not used by substrate
anyway)
   - Less dependencies for the clients in particular
   - Return type requires Clone in method call responses
   - Moved to tokio channels
   - Async subscription API (not used in this PR)

Major changes in this PR:
- The subscriptions are now bounded and if subscription can't keep up
with the server it is dropped
- CLI: add parameter to configure the jsonrpc server bounded message
buffer (default is 64)
- Add our own subscription helper to deal with the unbounded streams in
substrate

The most important things in this PR to review is the added helpers
functions in `substrate/client/rpc/src/utils.rs` and the rest is pretty
much chore.

Regarding the "bounded buffer limit" it may cause the server to handle
the JSON-RPC calls
slower than before.

The message size limit is bounded by "--rpc-response-size" thus "by
default 10MB * 64 = 640MB"
but the subscription message size is not covered by this limit and could
be capped as well.

Hopefully the last release prior to 1.0, sorry in advance for a big PR

Previous attempt: paritytech/substrate#13992

Resolves paritytech/polkadot-sdk#748, resolves
paritytech/polkadot-sdk#627
helin6 pushed a commit to boolnetwork/polkadot-sdk that referenced this issue Feb 5, 2024
* decode call with depth limit

* add tests

* taplo fmt
bgallois pushed a commit to duniter/duniter-polkadot-sdk that referenced this issue Mar 25, 2024
This is a rather big change in jsonrpsee, the major things in this bump
are:
- Server backpressure (the subscription impls are modified to deal with
that)
- Allow custom error types / return types (remove jsonrpsee::core::Error
and jsonrpee::core::CallError)
- Bug fixes (graceful shutdown in particular not used by substrate
anyway)
   - Less dependencies for the clients in particular
   - Return type requires Clone in method call responses
   - Moved to tokio channels
   - Async subscription API (not used in this PR)

Major changes in this PR:
- The subscriptions are now bounded and if subscription can't keep up
with the server it is dropped
- CLI: add parameter to configure the jsonrpc server bounded message
buffer (default is 64)
- Add our own subscription helper to deal with the unbounded streams in
substrate

The most important things in this PR to review is the added helpers
functions in `substrate/client/rpc/src/utils.rs` and the rest is pretty
much chore.

Regarding the "bounded buffer limit" it may cause the server to handle
the JSON-RPC calls
slower than before.

The message size limit is bounded by "--rpc-response-size" thus "by
default 10MB * 64 = 640MB"
but the subscription message size is not covered by this limit and could
be capped as well.

Hopefully the last release prior to 1.0, sorry in advance for a big PR

Previous attempt: paritytech/substrate#13992

Resolves paritytech#748, resolves
paritytech#627
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants