fix: Improve spec compliance by improve handling of responses without content-length or transfer-encoding #1540

evanderkoogh · 2022-07-12T10:38:22Z

       Otherwise, this is a response message without a declared message
       body length, so the message body length is determined by the
       number of octets received prior to the server closing the
       connection.

So now if we get a SocketClosed, we check if we have seen a content-length or transfer-encoding header and if not, we complete the message parsing.

…t-length or transfer-encoding fixes #1414 #1412 #1490

codecov-commenter · 2022-07-12T10:41:22Z

Codecov Report

Merging #1540 (ac2fbb9) into main (c485884) will decrease coverage by 0.01%.
The diff coverage is 87.50%.

@@            Coverage Diff             @@
##             main    #1540      +/-   ##
==========================================
- Coverage   94.90%   94.89%   -0.02%     
==========================================
  Files          50       50              
  Lines        4633     4638       +5     
==========================================
+ Hits         4397     4401       +4     
- Misses        236      237       +1

Impacted Files	Coverage Δ
lib/client.js	`97.64% <87.50%> (-0.12%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c485884...ac2fbb9. Read the comment docs.

ronag

Needs a test. Also I need to think about this. Feels like there might be some edge cases here.Especially with pipelining.

evanderkoogh · 2022-07-12T11:15:16Z

It does have a test: ac2fbb9#diff-ecc2f3b14c967f02fd074ac2d0e0b876e683c84f2755d251bbaa74a813a82130

It just doesn't include any test where transfer-encoding is set.. which doesn't make a ton of sense to write an entire test for, but easy enough to do.

evanderkoogh · 2022-07-12T11:37:08Z

Oh and the Node CI / Test / test (18, ubuntu-latest, experimental) (pull_request) Failing after 2m failing test is a timeout in some unrelated code.

Ahh.. I think I see where your concern is around the edge cases and pipelining.. I would need to check, but it could be indeed possible that when you are pipelining, it would error out the next request. I am going to investigate.

mcollina · 2022-07-12T12:35:41Z

I think we should just error all the pipelined requests. Given they are GETs, they will be picked up whenever another socket is connected.

mcollina

Could you add a test with more than one client.request() executing in parallel with pipelining enabled and disabled? So we verify this works ok in all situations.

guilledll · 2022-07-25T21:40:23Z

Hi everyone, any news about merging this? It will be very usefull for me to have it, thanks for the great work.

mcollina · 2022-07-26T08:25:06Z

This still needs one more test: #1540 (review)

@evanderkoogh are you still interesting in completing this PR?

kyranet · 2022-08-27T16:21:41Z

lib/client.js

@@ -615,6 +616,8 @@ class Parser {
      this.keepAlive += buf.toString()
    } else if (key.length === 14 && key.toString().toLowerCase() === 'content-length') {
      this.contentLength += buf.toString()
+    } else if (key.length === 16 && key.toString().toLowerCase() === 'transfer-encoding') {


Judging by the line 617 (since I'm not very familiar with Undici's code), shouldn't this be 17? transfer-encoding has 3 more characters than the previous header, and .length does indeed evaluate to 17.

Sounds like a test is missing for this?

ronag · 2022-08-27T17:07:52Z

lib/client.js

+  }
+
+  parser.destroy()
+  parser = null


This doesn't properly null the parser. Please don't use a local var.

madofo · 2022-11-10T14:46:35Z

Would be great to get this merged. Seems like @evanderkoogh has done most of the heavy lifting.

mcollina · 2022-11-10T19:42:08Z

Would you mind sending a PR finishing this?

This change somehow causes an lethal crash with SvelteKit sometimes. The root cause might be this one: nodejs/undici#1540

DamianGlowala · 2023-01-16T22:31:17Z

Affects me (a Nuxt user) as well; receiving an error when proxying the request through a Nitro server:

garand · 2023-03-24T13:35:31Z

😢 So is this not getting fix for now?

xgqfrms · 2023-09-15T04:51:33Z

Any updates?

This error sometimes occurs when I try to download some videos.

 async fetchVideo(url, filepath) {
    const filename = `${filepath}.mp4`;
    return fetch(url)
      // .then((res) => res.arrayBuffer())
      .then((res) => {
        console.log(`loading ... ⏳`);
        return res.arrayBuffer();
      })
      .then((buffer) => {
        console.log(`finished ✅`)
        let arrayBuffer = Buffer.from(buffer)
        fs.writeFile(filename, arrayBuffer, (err) => {
          if (err) {
            console.error(err);
            console.error(`❌ video url`, url);
          } else {
            console.log("video downloaded successfully");
          }
        });
      })
      .catch((err) => console.error(`❌❌ video url`, url, err));
      /*
        ❌❌ video url https://edu.xgqfrms.xyz/dc9bb2-1733219a4a8.mp4
        TypeError: terminated
          at Fetch.onAborted (node:internal/deps/undici/undici:11000:53)
          at Fetch.emit (node:events:513:28)
          at Fetch.terminate (node:internal/deps/undici/undici:10272:14)
          at Object.onError (node:internal/deps/undici/undici:11095:36)
          at Request.onError (node:internal/deps/undici/undici:6477:31)
          at errorRequest (node:internal/deps/undici/undici:8440:17)
          at TLSSocket.onSocketClose (node:internal/deps/undici/undici:7895:9)
          at TLSSocket.emit (node:events:525:35)
          at node:net:313:12
          at TCP.done (node:_tls_wrap:587:7) {
        [cause]: BodyTimeoutError: Body Timeout Error
            at Timeout.onParserTimeout [as _onTimeout] (node:internal/deps/undici/undici:7839:32)
            at listOnTimeout (node:internal/timers:566:11)
            at process.processTimers (node:internal/timers:507:7) {
          code: 'UND_ERR_BODY_TIMEOUT'
        }
      }
      */
  }

metcoder95 · 2023-09-15T07:48:25Z

PR has been closed for a while, but this seems to be still something to be considered to add. If interested, feel free to open a new PR with a proposal and we can outline something 🙂

evanderkoogh · 2023-09-16T12:16:01Z

The problem is that when I made that change, I fixed the actual issue, but then realised I introduced a bunch of edge-cases where there were still problems and I didn't have any bandwidth to figure out how to solve for all of those.

DamianGlowala · 2023-09-22T16:45:17Z

I would have assumed a non-compliance with RFC 7230 3.3.3 should gain a bit more attention from the Node.js team than simply 'feel free to open a new PR'...

Clearly this might be not a trivial problem to solve, which author of this PR already expressed in the above comment, hence any form of support from the team would be (very!) appreciated.

metcoder95 · 2023-09-25T07:52:40Z

The team has limited bandwidth for addressing all the issues opened at the time, hence we asked the community if they are willing to contribute to fixing the problem meanwhile we provide all the support and guidance through reviews.

Since the PR has been closed for a while, it might be better to open a new one where we can continue the efforts if someone volunteers 🙂

improve spec compliance by properly handling responses without conten…

ac2fbb9

…t-length or transfer-encoding fixes #1414 #1412 #1490

ronag reviewed Jul 12, 2022

View reviewed changes

evanderkoogh changed the title ~~Improve spec compliance by improve handling of responses without content-length or transfer-encoding~~ fix: Improve spec compliance by improve handling of responses without content-length or transfer-encoding Jul 12, 2022

mcollina requested changes Jul 14, 2022

View reviewed changes

Rich-Harris mentioned this pull request Jul 19, 2022

Fetch using load() in browser get response body but no response body returned during ssr sveltejs/kit#5606

Closed

ronag force-pushed the main branch from c0f0e0d to ecae314 Compare August 17, 2022 15:09

kyranet reviewed Aug 27, 2022

View reviewed changes

ronag requested changes Aug 27, 2022

View reviewed changes

lib/client.js

}

parser.destroy()

parser = null

Copy link

Member

ronag Aug 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't properly null the parser. Please don't use a local var.

Pankai222 mentioned this pull request Nov 10, 2022

Fetch fails when using SSR in dev environment streamchaser/streamchaser#325

Open

1 task

climba03003 mentioned this pull request Nov 17, 2022

internal undici:fetch throws error where node-fetch and curl succeed nodejs/node#45497

Closed

seanwu1105 added a commit to seanwu1105/news-traveler-client that referenced this pull request Nov 21, 2022

feat: show bias score on news component

c070d5d

This change somehow causes an lethal crash with SvelteKit sometimes. The root cause might be this one: nodejs/undici#1540

evanderkoogh closed this by deleting the head repository Mar 24, 2023

garand mentioned this pull request Mar 22, 2024

feat: use undici as fetch polyfill remix-run/remix#9106

Merged

2 tasks

jacob-ebey mentioned this pull request Mar 25, 2024

chore: validate responses can be consumed without a Content-Length or… #2995

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Improve spec compliance by improve handling of responses without content-length or transfer-encoding #1540

fix: Improve spec compliance by improve handling of responses without content-length or transfer-encoding #1540

evanderkoogh commented Jul 12, 2022

codecov-commenter commented Jul 12, 2022 •

edited

Loading

ronag left a comment

evanderkoogh commented Jul 12, 2022 •

edited

Loading

evanderkoogh commented Jul 12, 2022

mcollina commented Jul 12, 2022

mcollina left a comment

guilledll commented Jul 25, 2022

mcollina commented Jul 26, 2022

kyranet Aug 27, 2022

ronag Aug 27, 2022

ronag Aug 27, 2022

madofo commented Nov 10, 2022

mcollina commented Nov 10, 2022

DamianGlowala commented Jan 16, 2023

garand commented Mar 24, 2023

xgqfrms commented Sep 15, 2023 •

edited

Loading

metcoder95 commented Sep 15, 2023

evanderkoogh commented Sep 16, 2023

DamianGlowala commented Sep 22, 2023

metcoder95 commented Sep 25, 2023

fix: Improve spec compliance by improve handling of responses without content-length or transfer-encoding #1540

fix: Improve spec compliance by improve handling of responses without content-length or transfer-encoding #1540

Conversation

evanderkoogh commented Jul 12, 2022

codecov-commenter commented Jul 12, 2022 • edited Loading

Codecov Report

ronag left a comment

Choose a reason for hiding this comment

evanderkoogh commented Jul 12, 2022 • edited Loading

evanderkoogh commented Jul 12, 2022

mcollina commented Jul 12, 2022

mcollina left a comment

Choose a reason for hiding this comment

guilledll commented Jul 25, 2022

mcollina commented Jul 26, 2022

kyranet Aug 27, 2022

Choose a reason for hiding this comment

ronag Aug 27, 2022

Choose a reason for hiding this comment

ronag Aug 27, 2022

Choose a reason for hiding this comment

madofo commented Nov 10, 2022

mcollina commented Nov 10, 2022

DamianGlowala commented Jan 16, 2023

garand commented Mar 24, 2023

xgqfrms commented Sep 15, 2023 • edited Loading

Any updates?

metcoder95 commented Sep 15, 2023

evanderkoogh commented Sep 16, 2023

DamianGlowala commented Sep 22, 2023

metcoder95 commented Sep 25, 2023

codecov-commenter commented Jul 12, 2022 •

edited

Loading

evanderkoogh commented Jul 12, 2022 •

edited

Loading

xgqfrms commented Sep 15, 2023 •

edited

Loading