WebSocketProvider is not reconnecting to node after the connection is lost #7351

yanosh1982 · 2024-10-23T08:51:53Z

Hi, after this issue has been closed I've tried to set up a connection with WebSocketProvider on a node with reconnecting options, but the client is not reconnecting when the connection is lost. This is the snippet on how I am establishing the connection

 constructor(
        @Inject(WINSTON_MODULE_PROVIDER) 
        private readonly logger: LoggerService,
        private readonly appService: EventListenerService
    ) {
        try {
            const provider = new WebSocketProvider(process.env.BLOCKCHAIN_NODE_URL, {}, {autoReconnect: true, maxAttempts: 100000, delay: 5000});
            this.web3 = new Web3(provider);
            this.logger.log('info', `Connected to node: ${process.env.BLOCKCHAIN_NODE_URL}`);
        } catch (error) {
            this.logger.error(`Error connecting to Ethereum node: ${error}`);
            return;
        }
     }
  
  async onApplicationBootstrap() {
      this.logger.log('info', 'Bootstrapping %s', process.env.APP_NAME);

      await this.subscribeToNewBlockHeaders();
  }
  
  private async subscribeToNewBlockHeaders() {
    const subscription = await this.web3.eth.subscribe('newHeads', async (error, result) => {
      if (error) {
        this.logger.error(`Error subscribing to new block headers: ${error}`);
      } else {
        this.logger.debug(`New block header received: ${result.number}`);
      }
    });
  
    subscription.on("data", (blockHeader) => {
      this.logger.debug(`New block header received: ${blockHeader.number}`);
    });
  
    subscription.on("error", (error) => {
      this.logger.error(`Error subscribing to new block headers: ${error}`);
    });

    this.logger.log('info', 'Listening on %s events with subscription id %s', 'newHeads', subscription.id);
  
  };

this is the output

2024-10-23 08:32:13 [info]: Bootstrapping fdv-blockchain-event-listener 
2024-10-23 08:32:13 [info]: Listening on newHeads events with subscription id 0x1*
[Nest] 47500  - 10/23/2024, 8:32:13 AM     LOG [NestApplication] Nest application successfully started +58ms
2024-10-23 08:32:19 [debug]: New block header received: 9066189 
2024-10-23 08:32:25 [debug]: New block header received: 9066190 
2024-10-23 08:32:31 [debug]: New block header received: 9066191 
2024-10-23 08:32:37 [debug]: New block header received: 9066192 
2024-10-23 08:32:43 [debug]: New block header received: 9066193

If I turn of wi-fi, or stop and restart the node, the application simply stops logging without reconnecting.

Notice I'm using version 4.14.0 as shown in my package-lock.json

"node_modules/web3": {
      "version": "4.14.0",
      "resolved": "https://registry.npmjs.org/web3/-/web3-4.14.0.tgz",
      "integrity": "sha512-LohqxtSXXl4uA3abPK0bB91dziA5GygOLtO83p8bQzY+CYxp1fgGfiD8ahDRcu+WBttUhRFZmCsOhmrmP7HtTA==",
      "dependencies": {
        "web3-core": "^4.7.0",
        "web3-errors": "^1.3.0",
        "web3-eth": "^4.10.0",
        "web3-eth-abi": "^4.3.0",
        "web3-eth-accounts": "^4.2.1",
        "web3-eth-contract": "^4.7.0",
        "web3-eth-ens": "^4.4.0",
        "web3-eth-iban": "^4.0.7",
        "web3-eth-personal": "^4.1.0",
        "web3-net": "^4.1.0",
        "web3-providers-http": "^4.2.0",
        "web3-providers-ws": "^4.0.8",
        "web3-rpc-methods": "^1.3.0",
        "web3-rpc-providers": "^1.0.0-rc.2",
        "web3-types": "^1.8.1",
        "web3-utils": "^4.3.2",
        "web3-validator": "^2.0.6"
      }

Thank you

The text was updated successfully, but these errors were encountered:

luu-alex · 2024-10-23T14:39:48Z

@yanosh1982 hey there, can you join the web3.js chainsafe discord. i'll try to help there https://discord.gg/3KvND4Cx
im having trouble recreating the issue. It seems to be working fine with using node.js but ill try using nest

luu-alex · 2024-10-23T15:37:52Z

Edit: it seems to be provider dependent. I was able to get an error with closing geth. will keep you updated. thanks for reporting this

luu-alex · 2024-10-23T21:18:37Z

import { Injectable } from "@nestjs/common";
import Web3, { BlockHeaderOutput, WebSocketProvider } from 'web3';
import { NewHeadsSubscription } from "web3/lib/commonjs/eth.exports";

const gethURL = "ws://localhost:8545"
@Injectable()
export class Web3Service {
  private web3: Web3;
  subscription: NewHeadsSubscription | null;

  constructor() {
    // Initialize web3 with a provider
    this.web3 = new Web3(new WebSocketProvider(gethURL, {}, {
		delay: 500,
		autoReconnect: true,
		maxAttempts: 100,
	},));
  }

  getWeb3(): Web3 {
    return this.web3;
  }


    connectWeb3 = async (provider: WebSocketProvider) => {
        try {
            await new Promise((resolve, reject) => {
                provider.on('connect', (s) => {
                    this.resubscribeToEthNewBlockHeaders();
                //   ethereumLogger.info(`Connected to Ethereum node at ${wsUrl}`);
                resolve(true);
                });
                provider.on('error', (error: unknown) => {
                //   ethereumLogger.error(`Error while connecting to Ethereum node: ${error}`);
                console.log("connectweb3, cant connect");
                reject(error);
                });
            })
            return provider;
        } catch (error){
            console.log("error connecting to web3")
        }
    }

    subscribeToEthNewBlockHeaders = async (web3: Web3) => {
        if (this.subscription) {
            console.log("Already subscribed to block headers.");
            return;
          }
        this.subscription = await web3.eth.subscribe("newBlockHeaders", (error:
            unknown, result: unknown) => {
            if (error) {
                console.error("Error when subscribing to new block headers: ", error);
                return;
            }
            console.log("New block header: ", result);
        });
        this.subscription.on("data", (blockHeader: BlockHeaderOutput) => {
            console.log("New block header: ", blockHeader.number);
        })
        console.log("subscription created");

        this.subscription.on("error", (error: unknown) => {
            console.log("Error when subscribing to new block headers: ", error);
        })
        
        this.subscription.on("connected", () => {
            console.log("subscription connect")
        })
    }

    private async resubscribeToEthNewBlockHeaders(): Promise<void> {
        // Unsubscribe if an existing subscription exists
        if (this.subscription) {
          try {
            await this.subscription.unsubscribe();
            console.log("Previous subscription cancelled.");
          } catch (error) {
            console.error("Error when unsubscribing:", error);
          } finally {
            this.subscription = null; // Clear the subscription reference
          }
        }
    
        // Subscribe again
        await this.subscribeToEthNewBlockHeaders(this.web3);
      }
}


@Controller('web3')
export class Web3Controller {
    constructor(private readonly web3Service: Web3Service) {}

    @Get()
    getSubscription(): void {
        const web3 = this.web3Service.getWeb3();
        this.web3Service.connectWeb3(web3.currentProvider as WebSocketProvider);
        this.web3Service.subscribeToEthNewBlockHeaders(web3);
    }
}

Hey I was able to recreate your issue, where it connects but it seems to lose the subscription.
This issue is happening when the node restarts, the subscription is basically lost. In this code when we reconnect, we resubscribe and continue listening to events.
This is just a temporary fix,
Were you also facing issues when disconnecting from the internet or just when the provider resets?
Hope this helps, let me know if theres anything else I can help with. Otherwise ill continue creating a PR and i'll add a fix to the library so you don't need to go through all these hoops. Thanks!

yanosh1982 · 2024-10-24T08:43:21Z

Hey @luu-alex, thank you for your suggestion. This hasn't solved my problem. If i restart the node the events listening stops. My node is a hyperledger besu node and the network is a private proof of authority network

I think the whole problem depends on my application lifecycle. According to my needs i perform the connection onApplicationBootstrap that is fired, obviously, one time. My thought was that setting autoReconnect=true, the provider would ping the node to check the connection, performing a re-connection if it was lost. Check my code after importing your snippet

   private readonly contractAddress: string;
   private readonly contract: Contract<FdvDigestTrackerType>;
   private readonly web3: Web3
   private events: any[];

   subscription: NewHeadsSubscription | null;

   constructor(
       @Inject(WINSTON_MODULE_PROVIDER) 
       private readonly logger: LoggerService,
       private readonly appService: EventListenerService
   ) {
       try {
           const provider = new WebSocketProvider(process.env.BLOCKCHAIN_NODE_URL, {}, {
               delay: 500, // how long it takes between each reconnect attempt in ms
               autoReconnect: true, // default is true, false if you want to disable autoReconnect
               maxAttempts: 10, // default is 5 - number of reconnect attempts before giving up
           });
           this.web3 = new Web3(provider);
           this.logger.log('info', `Connected to node: ${process.env.BLOCKCHAIN_NODE_URL}`);
       } catch (error) {
           this.logger.error(`Error connecting to Ethereum node: ${error}`);
           return;
       }
       this.contractAddress = process.env.BLOCKCHAIN_FDV_DIGEST_TRACKER_CONTRACT_ADDRESS;
       const abi: FdvDigestTrackerType = FdvDigestTracker.abi as unknown as FdvDigestTrackerType;
       this.contract = new this.web3.eth.Contract(abi, this.contractAddress);

       this.events = [
           { name: 'NewDigest', event: this.contract.events.NewDigest },
           { name: 'DigestUpdated', event: this.contract.events.DigestUpdated },
           { name: 'DigestDeleted', event: this.contract.events.DigestDeleted }
       ];

   }

   async onApplicationBootstrap() {
       this.logger.log('info', 'Bootstrapping %s', process.env.APP_NAME);

       this.connectWeb3(this.web3.currentProvider as WebSocketProvider);

   }

   connectWeb3 = async (provider: WebSocketProvider) => {
       try {
           await new Promise((resolve, reject) => {
               provider.on('connect', (s) => {
                   this.resubscribeToEthNewBlockHeaders();
                   this.logger.log('info', `Connected to Ethereum node at ${process.env.BLOCKCHAIN_NODE_URL}`);
                   resolve(true);
               });
               provider.on('error', (error: unknown) => {
                   this.logger.error(`Error while connecting to Ethereum node: ${error}`);
                   reject(error);
               });
           })
           return provider;
       } catch (error){
           this.logger.error("error connecting to web3")
       }
   }

   subscribeToEthNewBlockHeaders = async (web3: Web3) => {
       if (this.subscription) {
           this.logger.warn("Already subscribed to block headers.");
           return;
         }
       this.subscription = await web3.eth.subscribe("newBlockHeaders", (error:
           unknown, result: unknown) => {
           if (error) {
               this.logger.error("Error when subscribing to new block headers: ", error);
               return;
           }
       });
       this.logger.log('info',"subscription created");
       
       this.subscription.on("data", (blockHeader: BlockHeaderOutput) => {
           this.logger.debug(`New block header: ${blockHeader.number}`);
       })

       this.subscription.on("error", (error: unknown) => {
           this.logger.error("Error when receiving newBlockHeaders data: ", error);
       })
       
       this.subscription.on("connected", () => {
           this.logger.log("subscription connect")
       })
   }

   private async resubscribeToEthNewBlockHeaders(): Promise<void> {
       // Unsubscribe if an existing subscription exists
       if (this.subscription) {
         try {
           await this.subscription.unsubscribe();
           this.logger.log('info',"Previous subscription cancelled.");
         } catch (error) {
           this.logger.log('info',"Error when unsubscribing:", error);
         } finally {
           this.subscription = null; // Clear the subscription reference
         }
       }
   
       // Subscribe again
       await this.subscribeToEthNewBlockHeaders(this.web3);
     }

If the node restarts, the application is bootstrapped and onApplicationBootstrap is not executed. The question is: the provider is still pinging the node and, in case, tries to reconnect to it?

luu-alex · 2024-10-26T18:59:14Z

will take a look into it

luu-alex · 2024-10-28T17:22:39Z

Update: I was able to confirm that it seems to be a hyperledger/besu issue with web3.js while other providers work when node restarts.

ill continue seeing how i can fix this

luu-alex · 2024-10-30T20:34:13Z

Hey there, heres an update with an extra listener With most providers when they are terminated they send an error but with besu they emit a disconnect. so we just need to cover that case

import { Injectable } from "@nestjs/common";
import Web3, { BlockHeaderOutput, WebSocketProvider } from 'web3';
import { NewHeadsSubscription } from "web3/lib/commonjs/eth.exports";

const gethURL = "ws://localhost:8545"
const besuURL = "ws://localhost:8546";
@Injectable()
export class Web3Service {
  private web3: Web3;
  subscription: NewHeadsSubscription | null;

  constructor() {
    // Initialize web3 with a provider
    // You can use HTTP provider, Websocket provider, or others depending on your use case
    this.web3 = new Web3(besuURL);
  }

  getWeb3(): Web3 {
    return this.web3;
  }


    connectWeb3 = async (provider: WebSocketProvider) => {

        try {
            
            await new Promise((resolve, reject) => {
                provider.on('connect', (s) => {
                    console.log('on connect in nest', s)
                    this.resubscribeToEthNewBlockHeaders();
                    console.log("connecting")
                //   ethereumLogger.info(`Connected to Ethereum node at ${wsUrl}`);
                resolve(true);
                });
                provider.on('error', (error: unknown) => {
                reject(error);
                });
                provider.on('close', () => {
                    this.resubscribeToEthNewBlockHeaders();
                })
                provider.on('end', (error: unknown) => {
                    reject(error);
                });
                provider.on('disconnect', () => {
                    this.resubscribeToEthNewBlockHeaders();
                })
            })
            return provider;
        } catch (error){
            console.log("error connecting to web3")
        }
    }

    subscribeToEthNewBlockHeaders = async (web3: Web3) => {
        if (this.subscription) {
            console.log("Already subscribed to block headers.");
            return;
          }
        this.subscription = await web3.eth.subscribe("newBlockHeaders", (error:
            unknown, result: unknown) => {
            if (error) {
                console.error("Error when subscribing to new block headers: ", error);
                return;
            }
            console.log("New block header: ", result);
        });
        console.log("error at subscribe to eth")
        this.subscription.on("data", (blockHeader: BlockHeaderOutput) => {
            console.log("New block header: ", blockHeader.number);
        })
        console.log("subscription created");

        this.subscription.on("error", (error: unknown) => {
            console.log("Error when subscribing to new block headers: ", error);
        })
        
        this.subscription.on("connected", () => {
            console.log("subscription connect")
        })
    }

    private async resubscribeToEthNewBlockHeaders(): Promise<void> {
        // Unsubscribe if an existing subscription exists
        if (this.subscription) {
          try {
            await this.subscription
            console.log("Previous subscription cancelled.");
          } catch (error) {
            console.error("Error when unsubscribing:", error);
          } finally {
            this.subscription = null; // Clear the subscription reference
          }
        }
    
        // Subscribe again
        await this.subscribeToEthNewBlockHeaders(this.web3);
      }
}

So in all, add this and it should work

provider.on('disconnect', () => {
    this.resubscribeToEthNewBlockHeaders();
})

yanosh1982 · 2024-10-30T23:40:52Z

Hey @luu-alex. I've tried your suggestion. This is my code now

connectWeb3 = async (provider: WebSocketProvider) => {
        try {
            await new Promise((resolve, reject) => {
                provider.on('connect', (s) => {
                    this.resubscribeToEthNewBlockHeaders();
                    this.logger.log('info', `Connected to Ethereum node at ${process.env.BLOCKCHAIN_NODE_URL}`);
                    resolve(true);
                });
                provider.on('error', (error: unknown) => {
                    this.logger.error(`Error while connecting to Ethereum node: ${error}`);
                    reject(error);
                });
                provider.on('disconnect', () => {
                    console.log("disconnect , connectweb3, cant connect");
                    this.resubscribeToEthNewBlockHeaders();
                })
            })
            return provider;
        } catch (error){
            this.logger.error("error connecting to web3")
        }
    }

When I stop the node, the disconnect event is fired and resubscribeToEthNewBlockHeaders function is called, but then I get an error when executing await this.subscription.unsubscribe(), and finally the service crashes

disconnect , connectweb3, cant connect
Error: connect ECONNREFUSED 172.24.181.127:8546
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1300:16)
    at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: 'myIpAddress',
  port: myPort
}

this is the complete function

private async resubscribeToEthNewBlockHeaders(): Promise<void> {
        // Unsubscribe if an existing subscription exists
        if (this.subscription) {
          try {
            await this.subscription.unsubscribe();
            this.logger.log('info',"Previous subscription cancelled.");
          } catch (error) {
            this.logger.log('info',"Error when unsubscribing:", error);
          } finally {
            this.subscription = null; // Clear the subscription reference
          }
        }
    
        // Subscribe again
        await this.subscribeToEthNewBlockHeaders(this.web3);
      }

Can't understand why the error is not caught in the catch block

luu-alex · 2024-10-31T00:34:45Z

private async resubscribeToEthNewBlockHeaders(): Promise<void> {
        // Unsubscribe if an existing subscription exists
        if (this.subscription) {
          try {
            await this.subscription
            console.log("Previous subscription cancelled.");
          } catch (error) {
            console.error("Error when unsubscribing:", error);
          } finally {
            this.subscription = null; // Clear the subscription reference
          }
        }
    
        // Subscribe again
        await this.subscribeToEthNewBlockHeaders(this.web3);
      }

ya not sure why too, this works on my end

luu-alex · 2024-10-31T17:44:39Z

Let me know if u face any other issues, thanks for reporting this

yanosh1982 changed the title ~~WebSocketProvider is not reconnecting on node after the connection is lost~~ WebSocketProvider is not reconnecting to node after the connection is lost Oct 23, 2024

yanosh1982 mentioned this issue Oct 23, 2024

WebSocket connection to Ethereum node closes/becomes idle after some time and reconnection fails to listen to smart contract events #6968

Closed

mconnelly8 added Bug Addressing a bug 4.x 4.0 related labels Oct 23, 2024

luu-alex self-assigned this Oct 23, 2024

luu-alex closed this as completed Oct 31, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WebSocketProvider is not reconnecting to node after the connection is lost #7351

WebSocketProvider is not reconnecting to node after the connection is lost #7351

yanosh1982 commented Oct 23, 2024 •

edited

Loading

luu-alex commented Oct 23, 2024 •

edited

Loading

luu-alex commented Oct 23, 2024 •

edited

Loading

luu-alex commented Oct 23, 2024 •

edited

Loading

yanosh1982 commented Oct 24, 2024 •

edited

Loading

luu-alex commented Oct 26, 2024

luu-alex commented Oct 28, 2024

luu-alex commented Oct 30, 2024 •

edited

Loading

yanosh1982 commented Oct 30, 2024

luu-alex commented Oct 31, 2024

luu-alex commented Oct 31, 2024

WebSocketProvider is not reconnecting to node after the connection is lost #7351

WebSocketProvider is not reconnecting to node after the connection is lost #7351

Comments

yanosh1982 commented Oct 23, 2024 • edited Loading

luu-alex commented Oct 23, 2024 • edited Loading

luu-alex commented Oct 23, 2024 • edited Loading

luu-alex commented Oct 23, 2024 • edited Loading

yanosh1982 commented Oct 24, 2024 • edited Loading

luu-alex commented Oct 26, 2024

luu-alex commented Oct 28, 2024

luu-alex commented Oct 30, 2024 • edited Loading

yanosh1982 commented Oct 30, 2024

luu-alex commented Oct 31, 2024

luu-alex commented Oct 31, 2024

yanosh1982 commented Oct 23, 2024 •

edited

Loading

luu-alex commented Oct 23, 2024 •

edited

Loading

luu-alex commented Oct 23, 2024 •

edited

Loading

luu-alex commented Oct 23, 2024 •

edited

Loading

yanosh1982 commented Oct 24, 2024 •

edited

Loading

luu-alex commented Oct 30, 2024 •

edited

Loading