test: add tests for invalid UTF-8 #40351

git-srinivas · 2021-10-06T19:27:55Z

make -j4 test (UNIX), or vcbuild test (Windows) passes
make lint passes

lpinca

// Run with --expose-internals flag

const assert = require('assert');
const { toUSVString } = require('internal/util');

const decoder = new TextDecoder();
const chunk = Buffer.from([0x66, 0x6f, 0x6f, 0xed, 0xa0, 0x80]); // foo + U+D800
const str = decoder.decode(chunk);

assert.strictEqual(toUSVString(str), 'foo\ufffd');

TextDecoder() already replaces each byte of the surrogate code point with U+FFFD.

lpinca · 2021-10-07T18:53:03Z

Hmm both Chrome and Safari work like this PR:

new Blob([new Uint8Array([0x66, 0x6f, 0x6f, 0xed, 0xa0, 0x80])]).text()
  .then((str) => str === 'foo\ufffd\ufffd\ufffd')
  .then(console.log);

but this is not consistent with

const { toUSVString } = require('internal/util');

toUSVString('foo\ud800'); // returns 'foo\ufffd';

git-srinivas · 2021-10-08T15:47:39Z

// Run with --expose-internals flag

const assert = require('assert');
const { toUSVString } = require('internal/util');

const decoder = new TextDecoder();
const chunk = Buffer.from([0x66, 0x6f, 0x6f, 0xed, 0xa0, 0x80]); // foo + U+D800
const str = decoder.decode(chunk);

assert.strictEqual(toUSVString(str), 'foo\ufffd');

TextDecoder() already replaces each byte of the surrogate code point with U+FFFD.

@lpinca Does it mean the issue 39804 is not valid?

ronag · 2021-10-08T15:56:46Z

```js
// Run with --expose-internals flag

const assert = require('assert');
const { toUSVString } = require('internal/util');

const decoder = new TextDecoder();
const chunk = Buffer.from([0x66, 0x6f, 0x6f, 0xed, 0xa0, 0x80]); // foo + U+D800
const str = decoder.decode(chunk);

assert.strictEqual(toUSVString(str), 'foo\ufffd');

TextDecoder() already replaces each byte of the surrogate code point with U+FFFD.

@lpinca Does it mean the issue 39804 is not valid?

Does your tests pass without the fix?

git-srinivas · 2021-10-11T07:48:33Z

yes. I ran test cases with --expose-internals flag. All of my test cases are passing without the fix.

ronag · 2021-10-11T08:12:10Z

Then I don't think this needs fixing. I would propose you remove the changes but keep the tests. We should still merge the tests.

git-srinivas · 2021-10-12T10:34:28Z

// Run with --expose-internals flag

const assert = require('assert');
const { toUSVString } = require('internal/util');

const decoder = new TextDecoder();
const chunk = Buffer.from([0x66, 0x6f, 0x6f, 0xed, 0xa0, 0x80]); // foo + U+D800
const str = decoder.decode(chunk);

assert.strictEqual(toUSVString(str), 'foo\ufffd');

TextDecoder() already replaces each byte of the surrogate code point with U+FFFD.

@lpinca @ronag
I am trying to understand this example to write some good testcases.
May i know how surrogate U+D800 is encoded as [0xed, 0xa0, 0x80]?

I tried with Buffer.from('\ud800','utf8') and what i get is <Buffer ef bf bd> .

lpinca · 2021-10-12T11:03:47Z

All surrogates are 3 bytes and in this range:

byte 1 = ED
byte 2 = [A0 - BF]
byte 3 = [80 - BF]

I tried with Buffer.from('\ud800','utf8') and what i get is <Buffer ef bf bd> .

That's because Buffer.from() replaces invalid UTF-8 byte sequences with U+FFFD.

Since the readable web streams already returns USVString The changes made to blob.js and consumers.js are reverted Test cases are added to check surrogate to USVString conversion for ReadablewebStreams Textdecoder and Blob PR-URL: nodejs#40351 Fixes: nodejs#39804

PR-URL: nodejs#40351 Fixes: nodejs#39804

lpinca · 2021-10-14T07:51:39Z

test/parallel/test-stream-consumers.js

+{
+  const passthrough = new PassThrough();
+
+  text(passthrough).then(common.mustCall(async (str) => {
+    assert.strictEqual(str.length, 11);
+    assert.deepStrictEqual(str, 'hellothere\ufffd');
+  }));
+
+  passthrough.write('hello');
+  setTimeout(() => passthrough.end('there\ud801'), 10);
+}


This does not make much sense because passthrough.write() and passthrough.end() will call Buffer.from() when the chunk is written. If anything, chunks should be Buffers or Uint8Arrays.

lpinca · 2021-10-14T07:53:34Z

test/parallel/test-whatwg-encoding-custom-textdecoder.js

+{
+  const decoder = new TextDecoder();
+  const chunk = Buffer.from('\ud807');
+  const str = decoder.decode(chunk);
+  assert.strictEqual(str, '\ufffd');
+}


Similar to the one above this does not make much sense to me. When decoder.decode(chunk) is called, chunk is already U+FFFD (valid UTF-8).

test/parallel/test-whatwg-encoding-custom-textdecoder.js

lpinca · 2021-10-22T08:49:36Z

@git-srinivas if you can please

Update PR title to update the actual scope of the PR
Squash commits and use a proper commit title and body as per point 1.

git-srinivas · 2021-10-22T08:59:52Z

Sure @lpinca I'll do the changes

lpinca · 2021-10-22T19:21:32Z

@git-srinivas can I suggest something like this?

test: add tests for invalid UTF-8

Verify that `Blob.prototype.text()`, `streamConsumers.text()` and
`TextDecoder.prototype.decode()` work as expected with invalid UTF-8.

Fixes: https://github.com/nodejs/node/issues/39804

I find the current commit message a bit misleading becausetoUSVString() works differently as per #40351 (comment).

Thank you.

Verify that `Blob.prototype.text()`, `streamConsumers.text()` and `TextDecoder.prototype.decode()` work as expected with invalid UTF-8. Fixes: nodejs#39804

nodejs-github-bot · 2021-11-11T18:02:47Z

CI: https://ci.nodejs.org/job/node-test-pull-request/40831/

nodejs-github-bot · 2021-11-12T19:08:45Z

CI: https://ci.nodejs.org/job/node-test-pull-request/40852/

nodejs-github-bot · 2021-11-13T06:30:13Z

CI: https://ci.nodejs.org/job/node-test-pull-request/40856/

nodejs-github-bot · 2021-11-13T09:30:46Z

CI: https://ci.nodejs.org/job/node-test-pull-request/40860/

nodejs-github-bot · 2021-11-13T10:28:51Z

CI: https://ci.nodejs.org/job/node-test-pull-request/40861/

nodejs-github-bot · 2021-11-13T16:44:55Z

CI: https://ci.nodejs.org/job/node-test-pull-request/40868/

nodejs-github-bot · 2021-11-14T06:05:10Z

CI: https://ci.nodejs.org/job/node-test-pull-request/40876/

nodejs-github-bot · 2021-11-14T17:37:01Z

CI: https://ci.nodejs.org/job/node-test-pull-request/40885/

nodejs-github-bot · 2021-11-15T16:43:25Z

CI: https://ci.nodejs.org/job/node-test-pull-request/40897/

Verify that `Blob.prototype.text()`, `streamConsumers.text()` and `TextDecoder.prototype.decode()` work as expected with invalid UTF-8. PR-URL: #40351 Fixes: #39804 Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Robert Nagy <ronagy@icloud.com>

lpinca · 2021-11-15T17:26:55Z

Landed in dc35aef.

Verify that `Blob.prototype.text()`, `streamConsumers.text()` and `TextDecoder.prototype.decode()` work as expected with invalid UTF-8. PR-URL: #40351 Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Robert Nagy <ronagy@icloud.com>

Verify that `Blob.prototype.text()`, `streamConsumers.text()` and `TextDecoder.prototype.decode()` work as expected with invalid UTF-8. PR-URL: nodejs#40351 Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Robert Nagy <ronagy@icloud.com>

Verify that `Blob.prototype.text()`, `streamConsumers.text()` and `TextDecoder.prototype.decode()` work as expected with invalid UTF-8. PR-URL: #40351 Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Robert Nagy <ronagy@icloud.com>

nodejs-github-bot added the needs-ci PRs that need a full CI run. label Oct 6, 2021

Mesteery added the stream Issues and PRs related to the stream subsystem. label Oct 6, 2021

git-srinivas force-pushed the body-mixin-text-to-return-usvstring branch from 0995716 to 18827d1 Compare October 7, 2021 02:57

ronag approved these changes Oct 7, 2021

View reviewed changes

jasnell approved these changes Oct 7, 2021

View reviewed changes

lpinca requested changes Oct 7, 2021

View reviewed changes

git-srinivas force-pushed the body-mixin-text-to-return-usvstring branch from 18827d1 to 5fa492a Compare October 13, 2021 15:08

git-srinivas added a commit to git-srinivas/node that referenced this pull request Oct 13, 2021

test: lint errors fixed in test cases

37ab7b0

PR-URL: nodejs#40351 Fixes: nodejs#39804

git-srinivas requested a review from lpinca October 14, 2021 06:04

lpinca reviewed Oct 14, 2021

View reviewed changes

git-srinivas force-pushed the body-mixin-text-to-return-usvstring branch from 4fb8c8a to e5857ba Compare October 21, 2021 19:45

git-srinivas requested a review from lpinca October 22, 2021 04:02

lpinca approved these changes Oct 22, 2021

View reviewed changes

test/parallel/test-whatwg-encoding-custom-textdecoder.js Outdated Show resolved Hide resolved

git-srinivas force-pushed the body-mixin-text-to-return-usvstring branch 2 times, most recently from 22de91f to 50b943b Compare October 22, 2021 12:16

git-srinivas changed the title ~~lib: body mixin text to return usvstring~~ test: ensure incomplete utf-8 byte sequence is converted to usv string Oct 22, 2021

cjihrig approved these changes Oct 22, 2021

View reviewed changes

test: add tests for invalid UTF-8

75c69af

Verify that `Blob.prototype.text()`, `streamConsumers.text()` and `TextDecoder.prototype.decode()` work as expected with invalid UTF-8. Fixes: nodejs#39804

git-srinivas force-pushed the body-mixin-text-to-return-usvstring branch from 50b943b to 75c69af Compare October 23, 2021 02:44

This was referenced Nov 9, 2021

CI Reliability 2021-11-09 nodejs/reliability#112

Open

CI Reliability 2021-11-10 nodejs/reliability#113

Open

CI Reliability 2021-11-11 nodejs/reliability#114

Open

github-actions bot mentioned this pull request Nov 12, 2021

CI Reliability 2021-11-12 nodejs/reliability#115

Open

35 tasks

github-actions bot mentioned this pull request Nov 13, 2021

CI Reliability 2021-11-13 nodejs/reliability#116

Open

31 tasks

github-actions bot mentioned this pull request Nov 14, 2021

CI Reliability 2021-11-14 nodejs/reliability#117

Open

41 tasks

github-actions bot mentioned this pull request Nov 15, 2021

CI Reliability 2021-11-15 nodejs/reliability#118

Open

44 tasks

lpinca closed this Nov 15, 2021

This was referenced Nov 16, 2021

CI Reliability 2021-11-16 nodejs/reliability#119

Open

CI Reliability 2021-11-17 nodejs/reliability#120

Open

CI Reliability 2021-11-18 nodejs/reliability#121

Open

CI Reliability 2021-11-19 nodejs/reliability#122

Open

targos mentioned this pull request Nov 26, 2021

v17.2.0 release proposal #40983

Merged

danielleadams mentioned this pull request Feb 1, 2022

v16.14.0 proposal #41804

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: add tests for invalid UTF-8 #40351

test: add tests for invalid UTF-8 #40351

git-srinivas commented Oct 6, 2021 •

edited

Loading

lpinca left a comment •

edited

Loading

lpinca commented Oct 7, 2021 •

edited

Loading

git-srinivas commented Oct 8, 2021

ronag commented Oct 8, 2021

git-srinivas commented Oct 11, 2021

ronag commented Oct 11, 2021

git-srinivas commented Oct 12, 2021

lpinca commented Oct 12, 2021 •

edited

Loading

lpinca Oct 14, 2021

lpinca Oct 14, 2021

lpinca commented Oct 22, 2021

git-srinivas commented Oct 22, 2021

lpinca commented Oct 22, 2021

nodejs-github-bot commented Nov 11, 2021

nodejs-github-bot commented Nov 12, 2021

nodejs-github-bot commented Nov 13, 2021

nodejs-github-bot commented Nov 13, 2021

nodejs-github-bot commented Nov 13, 2021

nodejs-github-bot commented Nov 13, 2021

nodejs-github-bot commented Nov 14, 2021

nodejs-github-bot commented Nov 14, 2021

nodejs-github-bot commented Nov 15, 2021

lpinca commented Nov 15, 2021 •

edited

Loading

test: add tests for invalid UTF-8 #40351

test: add tests for invalid UTF-8 #40351

Conversation

git-srinivas commented Oct 6, 2021 • edited Loading

lpinca left a comment • edited Loading

Choose a reason for hiding this comment

lpinca commented Oct 7, 2021 • edited Loading

git-srinivas commented Oct 8, 2021

ronag commented Oct 8, 2021

git-srinivas commented Oct 11, 2021

ronag commented Oct 11, 2021

git-srinivas commented Oct 12, 2021

lpinca commented Oct 12, 2021 • edited Loading

lpinca Oct 14, 2021

Choose a reason for hiding this comment

lpinca Oct 14, 2021

Choose a reason for hiding this comment

lpinca commented Oct 22, 2021

git-srinivas commented Oct 22, 2021

lpinca commented Oct 22, 2021

nodejs-github-bot commented Nov 11, 2021

nodejs-github-bot commented Nov 12, 2021

nodejs-github-bot commented Nov 13, 2021

nodejs-github-bot commented Nov 13, 2021

nodejs-github-bot commented Nov 13, 2021

nodejs-github-bot commented Nov 13, 2021

nodejs-github-bot commented Nov 14, 2021

nodejs-github-bot commented Nov 14, 2021

nodejs-github-bot commented Nov 15, 2021

lpinca commented Nov 15, 2021 • edited Loading

git-srinivas commented Oct 6, 2021 •

edited

Loading

lpinca left a comment •

edited

Loading

lpinca commented Oct 7, 2021 •

edited

Loading

lpinca commented Oct 12, 2021 •

edited

Loading

lpinca commented Nov 15, 2021 •

edited

Loading