Node request fails if response takes longer than 5 minutes (300s)

[clayreimann]

As documented in this issue Node's fetch implementation has a hard cap on timeout of 300s. It appears to be the case that their underlying nodejs http lib supports longer timeouts, up to infinity. However there is no way for me to configure that in the current node implementation.

See my comment for the results of my digging on this.

[jkaster]

the timeout configuration value for the Looker TypeScript SDK can be set in seconds. It can be configured for the whole SDK instance, or configured in the options parameter for a specific SDK message. It is applied in the base transport initRequest

[clayreimann]

@jkaster yes, and that doesn't matter when using the sdk from node when a request takes longer than 300 seconds to return¹. Looker's sdk-node implements the actual http call in nodeTransport using fetch. fetch, in node, exhibits this perplexing decision to return a failure if the server takes more than 300 seconds to send data. In my case this is manifested when attempting to run any of our longer running queries (that can take in excess of 20 minutes).

In my looker.ini file I have the following configuration that should (in theory) result in a 20 minute timeout:

[Looker]
base_url=https://MY_FQDN
client_id=MY_CLIENT_ID
client_secret=MY_CLIENT_SECRET
timeout=2400

However, in practice any query I try to run that exceeds the 300 second limit imposed by node's fetch function fails with an error like TypeError: fetch failed. And to make this harder to spot the node-sdk swallows the initial stack trace so my stumbling on the actual cause of this issue was more luck than anything.

¹ This is documented by numerous respondents on the referenced node issue as well as trivially demonstrated (at least for the v20.18.0 node on my mac) with the following node script.

// file: server.mjs
// node server.mjs
import http from 'http';

const server = http.createServer((req, res) => {
    // get timeout length from the path /timeout/:timeout
    // return 200 ok after the timeout
    const timeout = parseInt(req.url.split('/')[1]);
    console.log(`received request for timeout: ${timeout}`);
    setTimeout(() => {
        res.writeHead(200, { 'Content-Type': 'text/plain' });
        res.end('{"status": "ok", "timeout_sec": ' + timeout + '}');
    }, timeout * 1000);
});

server.listen(8080);
console.log('Server running at http://localhost:8080/');

setTimeout(() => {
    server.close();
}, 330 * 1000);

const done = new Promise((resolve) => {
    server.on('close', () => {
        resolve();
    });
});

async function makeCall(timeout) {
    const start = Date.now();
    try {
        const res = await fetch(`http://localhost:8080/${timeout}`);
        const json = await res.json();
        console.log(json);
    } catch (e) {
        console.error(`timeout ${timeout} errored after ${(Date.now() - start) / 1000}s`, e);
    }
}

Array.from(Array(330).keys()).forEach((sec) => {
    if (sec % 10 === 0 || (sec > 295 && sec < 305)) {
        makeCall(sec);
    }
})

await done;