Add Node.js httpAgent for FetchRequest options to enable proxy support

[ghost]

supersedes #2829

This is the PR to add an option for FetchRequest so that mimicking registerGetUrl function to only use proxies are no longer required ( See #4336 for the example code of it )

It will reduce needs for TypeScript projects to import and create bloat of types to fulfil the mimicked registerGetUrl function.

In short, only the following code will be required to use proxy connections

const { JsonRpcProvider, FetchRequest } = require('ethers');
const { HttpsProxyAgent } = require('https-proxy-agent');

const fetchReq = new FetchRequest(RPC_URL_HERE);
fetchReq.agent = new HttpsProxyAgent(PROXY_URL_HERE);
const provider = new JsonRpcProvider(fetchReq);

provider.getBlockNumber().then(console.log)

Without this PR, you may try this

const { JsonRpcProvider, FetchRequest } = require('ethers');
const { HttpsProxyAgent } = require('https-proxy-agent');
const http = require('node:http');
const { createProxy } = require('proxy');

const ETH_RPC = 'https://eth.llamarpc.com';
// Example that doesn't use proxy connections
const BSC_RPC = 'https://binance.llamarpc.com';
const HTTP_PROXY = `http://localhost:3128`;

/**
 * Define new FetchRequest used for a provider
 */
// Assigning custom getUrl function will apply to all ethers v6 providers

const fetchReq = new FetchRequest(ETH_RPC);
fetchReq.getUrlFunc = FetchRequest.createGetUrlFunc({ agent: new HttpsProxyAgent(HTTP_PROXY) });
const provider = new JsonRpcProvider(fetchReq);
const provider2 = new JsonRpcProvider(BSC_RPC);

let resolvePromise;
const promise = new Promise((resolve) => {resolvePromise = resolve});

promise.then(() => provider.getBlockNumber().then(console.log));
promise.then(() => provider2.getBlockNumber().then(console.log));

/**
 * (Optional) Define new http proxy server to demonstrate RPC proxy connection (like squid)
 */
const server = createProxy(http.createServer());

server.listen(3128, () => {
  resolvePromise();
  console.log('HTTP(s) proxy server listening on port %d', server.address().port);
});

server.on('connect', (res, socket) => {
  // This is where you could find out that ethers provider will connect RPC via proxy agent
  console.log(`Proxy connection from ${socket.remoteAddress} with headers: ${JSON.stringify(res.rawHeaders)}`);
});

[Mnemon1k]

supersedes #2829

This is the PR to add an option for FetchRequest so that mimicking registerGetUrl function to only use proxies are no longer required ( See #4336 for the example code of it )

It will reduce needs for TypeScript projects to import and create bloat of types to fulfil the mimicked registerGetUrl function.

In short, only the following code will be required to use proxy connections

const { JsonRpcProvider } = require('ethers');
const { HttpsProxyAgent } = require('https-proxy-agent');

FetchRequest.agent(new HttpsProxyAgent(PROXY_URL_HERE));
const provider = new JsonRpcProvider(RPC_URL_HERE);

provider.getBlockNumber().then(console.log)

In case if you don't want to mess up with the existing import

const FetchRequestProxy = FetchRequest.clone();
FetchRequestProxy.agent(new HttpsProxyAgent(PROXY_URL_HERE));
const provider = new JsonRpcProvider(new FetchRequestProxy(RPC_URL_HERE));

provider.getBlockNumber().then(console.log);

Hi I have an error
TypeError: FetchRequest.clone is not a function

"ethers": "6.7.1"

[ghost]

@Mnemon1k Hello, please try the following instead

const { ethers, JsonRpcProvider, FetchRequest } = require('ethers');
const { HttpsProxyAgent } = require('https-proxy-agent');
const fetch = require('cross-fetch');

const agent = new HttpsProxyAgent(PROXY_URL_HERE);

const getUrl = async (req, _signal) => {
  // Inherited from https://github.com/ethers-io/ethers.js/blob/main/src.ts/utils/geturl-browser.ts
  let signal;

  if (_signal) {
      const controller = new AbortController();
      signal = controller.signal;
      _signal.addListener(() => { controller.abort(); });
  }

  const init = {
    method: req.method,
    headers: req.headers,
    body: req.body || undefined,
    signal
  };

  // This is what we want
  init.agent = agent;

  // Inherited from https://github.com/ethers-io/ethers.js/blob/main/src.ts/utils/geturl-browser.ts
  const resp = await fetch(req.url, init);

  const headers = {};
  resp.headers.forEach((value, key) => {
    headers[key.toLowerCase()] = value;
  });

  const respBody = await resp.arrayBuffer();
  const body = (respBody == null) ? null: new Uint8Array(respBody);
  
  return {
    statusCode: resp.status,
    statusMessage: resp.statusText,
    headers,
    body
  };
};

/**
 * Define new FetchRequest used for a provider
 */
// Assigning custom getUrl function will apply to all ethers v6 providers
FetchRequest.registerGetUrl(getUrl);
const provider = new JsonRpcProvider(ETH_RPC);

const sleep = (sec) => new Promise(r => setTimeout(r, sec * 1000));

// Sleep until proxy server is booted up
sleep(2).then(() => provider.getBlockNumber().then(console.log));

It is kind of dirty since this is what the current ethers.js requires to do,

See the open issue #4353 for future improvements happening on.

[victoryeo]

the code snippet below, why it is able to work without a http proxy server running?

const fetchReq = new FetchRequest(ETH_RPC);
fetchReq.agent = new HttpsProxyAgent(HTTP_PROXY);
const provider = new JsonRpcProvider(fetchReq);
provider.getBlockNumber().then(console.log)

[ghost]

@victoryeo Right now you should modify global getUrl function in order to work with proxies

#4337 (comment)

The snippet you posted doesn't work right now without this PR

[ghost]

Closing the PR since an alternative approach is now available with the updated version

https://github.com/ethers-io/ethers.js/releases/tag/v6.8.0

For anyone interested please practice the latest example code

https://github.com/kaliubuntu0206/ethers-proxy-example/blob/main/index.js

[bergarces]

Closing the PR since an alternative approach is now available with the updated version

https://github.com/ethers-io/ethers.js/releases/tag/v6.8.0

For anyone interested please practice the latest example code

https://github.com/kaliubuntu0206/ethers-proxy-example/blob/main/index.js

The second link does no longer exist.