-
Notifications
You must be signed in to change notification settings - Fork 5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WebSocket connection to Ethereum node closes/becomes idle after some time and reconnection fails to listen to smart contract events #6968
Comments
Hi @royki , thanks for publishing this here! |
I am facing the same issue, could you solve it? |
I got the same issue. I try print the error log |
Hi @bugradursun, I solved it now using the |
Hi @SantiagoDevRel @jdevcs @avkos @Muhammad-Altabba @luu-alex |
@SantiagoDevRel Thank you for linking me this issue. I've tried to implement the solution reported above adjusting the code to my needs: import { Controller, Inject, LoggerService } from '@nestjs/common';
import { WINSTON_MODULE_PROVIDER } from 'nest-winston';
import Web3, { Contract, EventLog } from 'web3';
import { FdvDigestTrackerType } from "./types/fdv-digest-tracker.type";
import * as FdvDigestTracker from "./types/FdvDigestTracker.json";
import { EventListenerService } from './event-listener-service';
@Controller()
export class FdvDigestTrackerEventListener {
private readonly contractAddress: string;
private readonly contract: Contract<FdvDigestTrackerType>;
private readonly web3: Web3
private pingInterval: NodeJS.Timeout;
private events: any[];
constructor(
@Inject(WINSTON_MODULE_PROVIDER)
private readonly logger: LoggerService,
private readonly appService: EventListenerService
) {
try {
this.web3 = new Web3(new Web3.providers.WebsocketProvider(process.env.BLOCKCHAIN_NODE_URL));
this.logger.log('info', `Connected to node: ${process.env.BLOCKCHAIN_NODE_URL}`);
} catch (error) {
this.logger.error(`Error connecting to Ethereum node: ${error}`);
return;
}
this.contractAddress = process.env.BLOCKCHAIN_FDV_DIGEST_TRACKER_CONTRACT_ADDRESS;
const abi: FdvDigestTrackerType = FdvDigestTracker.abi as unknown as FdvDigestTrackerType;
this.contract = new this.web3.eth.Contract(abi, this.contractAddress);
this.events = [
{ name: 'NewDigest', event: this.contract.events.NewDigest },
{ name: 'DigestUpdated', event: this.contract.events.DigestUpdated },
{ name: 'DigestDeleted', event: this.contract.events.DigestDeleted }
];
}
async onApplicationBootstrap() {
this.logger.log('info', 'Bootstrapping %s', process.env.APP_NAME);
await this.subscribeToNewBlockHeaders();
this.startWebsocketPingInterval();
for (const event of this.events) {
await this.subscribe(event.name, event.event(), this.onData);
}
this.web3.currentProvider.on('error', async error => {
this.logger.error(`Websocket Error: ${error}`);
this.cleanup();
this.startWebsocketPingInterval();
});
this.web3.currentProvider.on('end', async error => {
this.logger.error(`Websocket connection ended: ${error}`);
this.cleanup();
this.startWebsocketPingInterval();
})
}
private async startWebsocketPingInterval() {
this.pingInterval = setInterval(async () => {
try {
await this.web3.eth.net.isListening();
this.logger.log('info', `Websocket connection alive (ping successful)`);
} catch (error) {
this.logger.warn(`Ping failed, connection might be inactive, ${error}`);
await this.resetProviderAndResubscribe();
}
}, parseInt(process.env.BLOCKCHAIN_WS_PROVIDER_MS_PING_INTERVAL));
}
private cleanup() {
clearInterval(this.pingInterval);
}
private onData = async (event: EventLog) => {
this.logger.log('info', '%s: Processing Event of type %s: %o', event.returnValues.eventId, event.event, event);
const processResult = await this.appService.processEvent(event);
if(processResult){
this.logger.log('info', '%s: Event %s processed successfully', event.returnValues.eventId, event.returnValues.eventId);
} else {
this.logger.log('info', '%s: Event %s processed with errors', event.returnValues.eventId, event.returnValues.eventId);
}
}
private async subscribeToNewBlockHeaders() {
const subscription = await this.web3.eth.subscribe('newHeads', async (error, result) => {
if (error) {
this.logger.error(`Error subscribing to new block headers: ${error}`);
} else {
this.logger.log('info', `New block header received: ${result.number}`);
}
});
subscription.on("data", (blockHeader) => {
this.logger.log('info', `New block header received: ${blockHeader.number}`);
});
subscription.on("error", (error) => {
this.logger.error(`Error subscribing to new block headers: ${error}`);
});
this.logger.log('info', 'Listening on %s events with subscription id %s', 'newHeads', subscription.id);
};
private async resetProviderAndResubscribe() {
// Reset provider
this.web3.setProvider(new Web3.providers.WebsocketProvider(process.env.BLOCKCHAIN_NODE_URL));
this.logger.log('info', `ReConnected to node:: ${process.env.BLOCKCHAIN_NODE_URL}`);
await this.subscribeToNewBlockHeaders();
for (const event of this.events) {
await this.subscribe(event.name, event.event(), this.onData);
}
}
private async subscribe(eventName: string, subscription: any, onData: (event: EventLog) => void) {
subscription.on("connected", (subscriptionId) => {
this.logger.log('info', 'Listening on %s events with subscription id %s on contract %s', eventName, subscriptionId, process.env.BLOCKCHAIN_FDV_DIGEST_TRACKER_CONTRACT_ADDRESS);
})
subscription.on('data', onData);
subscription.on('error', (error) => {
this.logger.error('Error subscribing %s events: %o', eventName, error);
});
}
} and now events are processed. Anyway I've tried to test locally what happens when the ping to the network fails turning off the wifi connection. The process is idle until the connection is back and then I obtain the following output: 2024-06-18 06:50:51 [info]: Bootstrapping fdv-blockchain-event-listener
2024-06-18 06:50:51 [info]: **Listening on newHeads events with subscription id 0x106**
[Nest] 82077 - 06/18/2024, 6:50:51 AM LOG [NestApplication] Nest application successfully started +57ms
2024-06-18 06:50:51 [info]: Listening on NewDigest events with subscription id 0x107 on contract 0x5bcf3FB71dbc54a0eaf1F414911538b5142D9b22
2024-06-18 06:50:51 [info]: Listening on DigestUpdated events with subscription id 0x108 on contract 0x5bcf3FB71dbc54a0eaf1F414911538b5142D9b22
2024-06-18 06:50:51 [info]: Listening on DigestDeleted events with subscription id 0x109 on contract 0x5bcf3FB71dbc54a0eaf1F414911538b5142D9b22
2024-06-18 06:50:52 [info]: New block header received: 7820125
2024-06-18 06:51:14 [error]: Websocket Error: PendingRequestsOnReconnectingError: CONNECTION ERROR: Provider started to reconnect before the response got received!
2024-06-18 06:51:14 [error]: Websocket Error: PendingRequestsOnReconnectingError: CONNECTION ERROR: Provider started to reconnect before the response got received!
2024-06-18 06:51:14 [error]: Websocket Error: PendingRequestsOnReconnectingError: CONNECTION ERROR: Provider started to reconnect before the response got received!
2024-06-18 06:51:14 [error]: Websocket Error: PendingRequestsOnReconnectingError: CONNECTION ERROR: Provider started to reconnect before the response got received!
2024-06-18 06:51:14 [warn]: Ping failed, connection might be inactive, PendingRequestsOnReconnectingError: CONNECTION ERROR: Provider started to reconnect before the response got received!
2024-06-18 06:51:14 [info]: ReConnected to node:: ws://172.24.181.127:8546
2024-06-18 06:51:14 [warn]: Ping failed, connection might be inactive, PendingRequestsOnReconnectingError: CONNECTION ERROR: Provider started to reconnect before the response got received!
2024-06-18 06:51:14 [info]: ReConnected to node:: ws://172.24.181.127:8546
2024-06-18 06:51:14 [warn]: Ping failed, connection might be inactive, PendingRequestsOnReconnectingError: CONNECTION ERROR: Provider started to reconnect before the response got received!
2024-06-18 06:51:14 [info]: ReConnected to node:: ws://172.24.181.127:8546
2024-06-18 06:51:14 [warn]: Ping failed, connection might be inactive, PendingRequestsOnReconnectingError: CONNECTION ERROR: Provider started to reconnect before the response got received!
2024-06-18 06:51:14 [info]: ReConnected to node:: ws://172.24.181.127:8546
2024-06-18 06:51:14 [info]: Listening on newHeads events with subscription id 0x10a
2024-06-18 06:51:14 [info]: Listening on newHeads events with subscription id 0x10b
2024-06-18 06:51:14 [info]: Listening on newHeads events with subscription id 0x10d
2024-06-18 06:51:14 [info]: Listening on newHeads events with subscription id 0x10c
/home/node/app/node_modules/web3-core/src/web3_request_manager.ts:362
throw new InvalidResponseError<ErrorType, RequestType>(response, payload);
^
InvalidResponseError: Returned error: Subscription not found
at Web3RequestManager._processJsonRpcResponse (/home/node/app/node_modules/web3-core/src/web3_request_manager.ts:362:11)
at Web3RequestManager.<anonymous> (/home/node/app/node_modules/web3-core/src/web3_request_manager.ts:206:16)
at Generator.next (<anonymous>)
at fulfilled (/home/node/app/node_modules/web3-core/lib/commonjs/web3_request_manager.js:21:58) {
innerError: { code: -32000, message: 'Subscription not found' },
code: 101,
data: undefined,
request: {
jsonrpc: '2.0',
id: '2a57c01e-9633-452a-9294-7020e2e6fca1',
method: 'eth_unsubscribe',
**params: [ '0x106' ]**
}
}
Waiting for the debugger to disconnect... Please notice that logs entries are repeated a variable number of times and the process terminate trying to unsubscribe the initial newHeads subscription with id 0x106. This differs from @royki provided output that looks like the following 2024-04-11T23:49:58.404Z [ERROR]: Websocket Error: PendingRequestsOnReconnectingError: CONNECTION ERROR: Provider started to reconnect before the response got received!
2024-04-11T23:49:58.406Z [WARN]: Ping failed, connection might be inactive, PendingRequestsOnReconnectingError: CONNECTION ERROR: Provider started to reconnect before the response got received!
2024-04-11T23:49:58.415Z [INFO]: ReConnected to EnergyWebChain node:: wss://xxxxxxxx/ws Can someone help me to solve this problem? Thank you |
Hi @royki sorry for the long wait, will dive into this now |
@royki @yanosh1982 this issue seems to be from an improper error handling within our ws providers. this will be fixed next release. thanks so much for the issue report. |
Pr has been merged, these changes will be available in next release. You shouldn't need all this error checking now :). I'll update this comment when we do the next release. Thank you! |
Hi @luu-alex, thank you!!! Can you please provide a snippet to perform a web socket connection with reconnection options after your patch? Regards! |
@yanosh1982 yes i can! i'll do it sometime soon, the patch has been released so let me knwo if u run into other issues. thanks! |
Hi @luu-alex. Some time ago I asked you to provide an example on how to perform a ws connection with reconnection options after your patch. Can you kindly provide some snippet? Thank you. Regards |
cc: @mconnelly8 |
Sorry for the long awaited response. heres a code snippet. it was a bug fix, so what should have been working in the past should work now, there is no changes in terms of code |
Hi @luu-alex. Thank you for your snippet. I've compared it my implementation reported in this issue I've opened this morning. In my case I'm running the code in NestJS but It should work the same. |
Expected behavior
The WebSocket connection to the Ethereum node should remain active. The connection should continuously listen to smart contract events.
Actual behavior
The WebSocket connection to the Ethereum node either closes or becomes idle after some time (approximately 5-6 hours or 10 hours). When this happens, the reconnection attempts fail to listen to smart contract events.
Steps to reproduce the behavior
Subscribe to new block headers and smart contract events.
subscribeToEthNewBlockHeaders
.To address this issue, here are some resources to explore:
Here is code that temporarily solves the issue by incorporating some of the suggestions found above, such as pinging, set provider, and reconnect listener.
Logs
Environment
Volta
andEnergyWebChain
v18.12.0 LTS
or higherv9.8.1
"^4.4.0"
Ubuntu 22.04 LTS
The text was updated successfully, but these errors were encountered: