Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ResourceExhausted error while running the Node Audit #177

Closed
remnantdev opened this issue Dec 7, 2023 · 3 comments · Fixed by #178
Closed

ResourceExhausted error while running the Node Audit #177

remnantdev opened this issue Dec 7, 2023 · 3 comments · Fixed by #178

Comments

@remnantdev
Copy link

We run into a ResourceExhausted error while running the NodeAudit report.

Error received from peer ipv4:10.9.82.14:8465","file":"/var/local/git/grpc/src/core/lib/surface/call.cc","file_line":1075,"grpc_message":"grpc: received message larger than max (214449486 vs. 209715200)","grpc_status":8}
  • The error does not seem to be dependant on the requested time range.
  • We run the report with DisableFiat:true and UnknownGranularity.
  • This error occurs on only one of our nodes presumably the one with the most onchain transactions.
    • We do use our LND node to send and receive onchain transactions
    • Looking through the report code, it seems that all transactions are requested from the LND Server in a single RPC call which could be over the maxMsgRecvSize of 200MB and would explain it occurs no matter what the requested time range.

Expected behavior

Node Audit should produce the report.

Actual behavior

Node Audit should produce the report

To reproduce

This error now occurs systematically on one node when we run Node Audit

System information

  • Connecting through gRPC
  • Version: Faraday v0.2.11-alpha
@guggero
Copy link
Member

guggero commented Dec 7, 2023

Hey, thanks for the report. You are correct, the receive size is too small. We should really start paginating calls to ListTransactions, but in the meantime, can you please try with #178?

@remnantdev
Copy link
Author

We have deployed a patched version and the report ran successfully!

On the first run we received a deadline exceeded error. We might also want to increase the default deadline.

@guggero
Copy link
Member

guggero commented Dec 7, 2023

I pushed a new commit that adds the --lnd.requesttimeout config options that takes a duration. So you can bump it to --lnd.requesttimeout=2m or whatever you feel is appropriate (note the time unit suffix, without a unit I believe this is in nano-seconds).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants