fix(MockHttpSocket): exhaust .write() callbacks for mocked requests #542

kettanaito · 2024-03-31T10:43:10Z

The socket now correctly emits the finish event for mocked responses.
The socket now calls all the .write() callbacks for mocked responses.
The socket now correctly emits the end event when the readable ends.
The socket now flushes all the write calls (callbacks) if someone starts reading this.requestStream ReadableStream. We still flush any present write calls in .respondWith() (nothing could've read the request stream up until that point).

Todos

Fix the event order: fix(MockHttpSocket): exhaust .write() callbacks for mocked requests #542 (comment)

kettanaito · 2024-03-31T10:44:13Z

src/interceptors/ClientRequest/MockHttpSocket.ts

+    // It has been used by the request parser to construct
+    // a Fetch API Request instance representing this request.
+    for (const [_, __, writeCallback] of this.writeBuffer) {
+      writeCallback?.()


The actual fix.

When ClientRequest writes the data to the socket, the .write() method of the MockSocket just buffers the chunks in this.writeBuffer. It never actually does any writes (we cannot do that until we know if the connection is possible) and so it never called any write callbacks (such as the onFinish callback).

Maybe we now can/should return false in the mocked socket.write?

Now I see we are calling the write callback after the response, which causes a deadlock because not all data is ready for the user:

const http = require('http') const sinon = require('sinon') const { ClientRequestInterceptor } = require('@mswjs/interceptors/ClientRequest') const interceptor = new ClientRequestInterceptor() interceptor.on('request', async function rootListener({ request }) { console.log('before'); await request.arrayBuffer() console.log('never get here'); request.respondWith(new Response('OK!')) }) interceptor.apply() const reqWriteCallback = sinon.spy() const req = http.request( { host: 'example.com', method: 'POST', path: '/', port: 80, }, res => { console.log(2) res.on('end', () => { console.log(3); }) // Streams start in 'paused' mode and must be started. // See https://nodejs.org/api/stream.html#stream_class_stream_readable res.resume() }, ) req.write('mamma mia', null, () => { console.log(1); reqWriteCallback() req.end() })

Are you talking about the response event when you mention "response"?

I mean we call the write callback inside the responseWith function. sorry for the confusion.

Is that the expected order?

TL;DR: I think yes.

It's a design choice. If I understand you correctly, you would rather the whole request body to be ready for the user in the interceptor so that they can do await request.arrayBuffer without a fear of deadlock like this.
We can also say that the body isn't guaranteed to be ready, and they user needs to take care of this scenario by itself (I think they can with request.arrayBuffer().then(..))

Actually, not quite. The request event is emitted as soon as the request's headers are sent. So the request body may still be streaming. If the user decides to read it, whichever body reading method will return a Promise, and the entire request listener would have to wait for that promise.

I wonder if that conflicts with the write callbacks. As in, in order to read the request body, the socket has to call the callbacks of .write(), and we are calling those only as a part of respondWith().

That shouldn't be the case though. The "request body" the user is reading in the request listener is our internal request body buffer where we buffer the chunks pushed.

That shouldn't be the case though. The "request body" the user is reading in the request listener is our internal request body buffer where we buffer the chunks pushed.

But the internal request body never ends because we never call the write callback, which ends the request body. (we never do: this.requestStream.push(null))

Not sure we should be doing this. Need to look more into this behavior but it's the actual request that calls .end() on the writable stream. Perhaps the issue is that we are not translating that to this.requestStream.push(null).

I've pushed a fix in 7be0242.

Basically, if something attempts to read the request buffer (e.g. await request.text() in the interceptor), the MockHttpSocket will immediately start flushing all the write callbacks.

If nothing reads the buffer, the write callbacks will be flushed before the mocked response starts streaming or before the original request is made.

@mikicho, can you please take a look if this fixes your use case? I put it in the test also with that commit.

kettanaito · 2024-03-31T10:44:28Z

src/interceptors/ClientRequest/MockHttpSocket.ts

@@ -187,6 +188,7 @@ export class MockHttpSocket extends MockSocket {
      .on('prefinish', () => this.emit('prefinish'))
      .on('finish', () => this.emit('finish'))
      .on('close', (hadError) => this.emit('close', hadError))
+      .on('end', () => this.emit('end'))


We never forwarded the end event of the original socket. I think this should be added.

kettanaito · 2024-03-31T10:46:25Z

src/interceptors/Socket/MockSocket.ts

-    }
-
-    return true
+    return super.push(chunk, encoding)


Call super.push() for the .push(null) to correctly transition the socket into the end state:

emit end event

set readable to false

set readableEnded to true

See: https://github.com/nodejs/node/blob/3a456c6db802b5b25594d3a9d235d4989e9f7829/lib/internal/streams/readable.js#L1696

kettanaito · 2024-03-31T10:47:12Z

src/interceptors/Socket/MockSocket.ts

@@ -22,6 +22,10 @@ export class MockSocket extends net.Socket {
    super()
    this.connecting = false
    this.connect()
+
+    this._final = (callback) => {
+      callback(null)


Implement the _final method so socket writes transition the socket to the right "final" state:

set writableFinished to true

emit finish event

See: https://github.com/nodejs/node/blob/3a456c6db802b5b25594d3a9d235d4989e9f7829/lib/internal/streams/writable.js#L907

this._final() is called on prefinish.

kettanaito · 2024-03-31T10:47:34Z

test/modules/http/compliance/http-req-write.test.ts

+  expect(finishListener).toHaveBeenCalledTimes(1)
+})
+
+it('emits "finish" for a mocked request', async () => {


@mikicho, this is the failing write finish test you sent me. It's passing now.

kettanaito · 2024-03-31T10:47:58Z

src/interceptors/Socket/MockSocket.test.ts

+
+  // Read the data to free the buffer and
+  // make Socket emit "end".
+  socket.read()


This was a broken test. We have to call .read() to exhaust the readable stream so it emits the end event.

kettanaito · 2024-03-31T10:49:30Z

Hi, @mikicho 👋 This is the fix for the finish event not being emitted for mocked requests. Can you please give it a look and see if you spot anything off? Thanks.

…et-write

mikicho · 2024-03-31T19:16:10Z

I just tested it, and it fixed some tests, YaY.
However, I encountered an interesting time-based edge case that our tests caught.
I'm not sure if this is the expected behavior, but somehow, Nock preserves the same event order as Node.
Anyway, if you think about something quickly, it should be cool; if not, we may revisit it later.

Scenario A - quick abort:

const http = require('http')
const sinon = require('sinon')
const { ClientRequestInterceptor } = require('@mswjs/interceptors/ClientRequest')

const interceptor = new ClientRequestInterceptor()

interceptor.on('request', async function rootListener({ request }) {
  request.respondWith(new Response('OK!'))
})
// interceptor.apply()

const req = http.request('http://example.com')
const emitSpy = sinon.spy(req, 'emit')

req.on('finish', () => {
  setTimeout(() => {
    req.abort()
  }, 10)
})
req.end()

setTimeout(() => {
  console.log('unmocked');
  console.log(emitSpy.args.map(i => i[0]));
}, 500)

➜ nock git:(Michael/fetch-support) ✗ node a.js
unmocked
[ 'socket', 'prefinish', 'finish', 'response', 'close', 'abort' ]
➜ nock git:(Michael/fetch-support) ✗ node a.js
mocked
[ 'socket', 'prefinish', 'finish', 'response', 'close', 'abort' ]

Scenario B - abort after long time:

const http = require('http')
const sinon = require('sinon')
const { ClientRequestInterceptor } = require('@mswjs/interceptors/ClientRequest')

const interceptor = new ClientRequestInterceptor()

interceptor.on('request', async function rootListener({ request }) {
  request.respondWith(new Response('OK!'))
})
// interceptor.apply()

const req = http.request('http://example.com')
const emitSpy = sinon.spy(req, 'emit')

req.on('finish', () => {
  setTimeout(() => {
    req.abort()
  }, 1000)
})
req.on('error', err => {
  // console.log(err);
})
req.end()

setTimeout(() => {
  console.log('unmocked');
  console.log(emitSpy.args.map(i => i[0]));
}, 5000)

➜ nock git:(Michael/fetch-support) ✗ node a.js
mocked
[ 'socket', 'prefinish', 'finish', 'response', 'close', 'abort' ]
➜ nock git:(Michael/fetch-support) ✗ node a.js
unmocked
[ 'socket', 'prefinish', 'finish', 'abort', 'error', 'close' ]
➜ nock git:(Michael/fetch-support) ✗

For small abort timeout, we emit response event that Node.js doesn't.

kettanaito · 2024-03-31T20:35:03Z

@mikicho, we should improve event compatibility but it's out of scope of this change. The order of abort/error/close events is also off compared to Node.js. Let's tackle those separately.

For now, some XHR tests are failing and I need to look at why. Fail in the base feature branch also, so unlikely to be relevant to these changes.

mikicho · 2024-03-31T20:39:43Z

@kettanaito Agreed.

For now, some XHR tests are failing and I need to look at why. Fail in the base feature branch also, so unlikely to be relevant to these changes.

I saw this failing test and expected them to reflect in one of Nock's, but it didn't (although I skipped on around ~40 for now). It surprised me a little.

Can you please take a look at this before merging this PR? #542 (comment)

…et-write

kettanaito · 2024-03-31T21:43:22Z

The failing test has been resolved in the base feature branch. I forgot to migrate the new interceptor to use a fixed request ID header vs the hard-coded x-request-id string, which broke the XHR -> ClientRequest interceptors deduplication and resulted in extra request/response events on the interceptor.

kettanaito · 2024-04-12T19:40:23Z

src/interceptors/ClientRequest/MockHttpSocket.ts

+          // flush the write buffer to trigger the callbacks.
+          // This way, if the request stream ends in the write callback,
+          // it will indeed end correctly.
+          this.flushWriteBuffer()


If someone starts reading the request stream, flush all the write calls immediately. This way, if the stream ends in the write callback, it will end correctly:

interceptor.on('request', async ({ request }) => { // Start reading the request stream before // deciding whether to mock it or not. // This triggers "read()" on the request's ReadableStream. await request.arrayBuffer() }) req.write('foo', () => req.end()

mikicho · 2024-04-12T21:08:17Z

It fixed more tests!

(P.S.: Some tests have revealed that Nock is not behaving properly, and the new interceptor has found the issue!)

kettanaito · 2024-04-12T21:18:23Z

That is the best kind of discovery! Yohoo! Should we merge this?

mikicho · 2024-04-12T21:22:15Z

kettanaito added 2 commits March 31, 2024 12:42

fix(MockSocket): implement "_final", rely on "super.push()"

d6ea014

fix(MockHttpSocket): exhaust .write() callbacks for mocked requests

10816a0

kettanaito changed the base branch from main to feat/yet-another-socket-interceptor March 31, 2024 10:43

kettanaito commented Mar 31, 2024

View reviewed changes

kettanaito mentioned this pull request Mar 31, 2024

feat(ClientRequest): use net.Socket interceptor #515

Merged

18 tasks

Merge branch 'feat/yet-another-socket-interceptor' into fix/mock-sock…

469f481

…et-write

Merge branch 'feat/yet-another-socket-interceptor' into fix/mock-sock…

c6f1066

…et-write

fix(MockHttpSocket): flush write buffer on "requestStream" read

7be0242

kettanaito commented Apr 12, 2024

View reviewed changes

kettanaito merged commit 824998b into feat/yet-another-socket-interceptor Apr 12, 2024
1 check passed

kettanaito deleted the fix/mock-socket-write branch April 12, 2024 21:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(MockHttpSocket): exhaust .write() callbacks for mocked requests #542

fix(MockHttpSocket): exhaust .write() callbacks for mocked requests #542

kettanaito commented Mar 31, 2024 •

edited

Loading

kettanaito Mar 31, 2024

mikicho Mar 31, 2024 •

edited

Loading

mikicho Mar 31, 2024 •

edited

Loading

kettanaito Mar 31, 2024

mikicho Mar 31, 2024

mikicho Mar 31, 2024

kettanaito Mar 31, 2024

mikicho Mar 31, 2024

kettanaito Apr 2, 2024

kettanaito Apr 12, 2024 •

edited

Loading

kettanaito Mar 31, 2024

kettanaito Mar 31, 2024

kettanaito Mar 31, 2024

kettanaito Mar 31, 2024

kettanaito Mar 31, 2024

kettanaito commented Mar 31, 2024

mikicho commented Mar 31, 2024

kettanaito commented Mar 31, 2024

mikicho commented Mar 31, 2024

kettanaito commented Mar 31, 2024

kettanaito Apr 12, 2024

mikicho commented Apr 12, 2024

kettanaito commented Apr 12, 2024

mikicho commented Apr 12, 2024

fix(MockHttpSocket): exhaust .write() callbacks for mocked requests #542

fix(MockHttpSocket): exhaust .write() callbacks for mocked requests #542

Conversation

kettanaito commented Mar 31, 2024 • edited Loading

Todos

Choose a reason for hiding this comment

mikicho Mar 31, 2024 • edited Loading

Choose a reason for hiding this comment

mikicho Mar 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kettanaito Apr 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kettanaito commented Mar 31, 2024

mikicho commented Mar 31, 2024

kettanaito commented Mar 31, 2024

mikicho commented Mar 31, 2024

kettanaito commented Mar 31, 2024

Choose a reason for hiding this comment

mikicho commented Apr 12, 2024

kettanaito commented Apr 12, 2024

mikicho commented Apr 12, 2024

kettanaito commented Mar 31, 2024 •

edited

Loading

mikicho Mar 31, 2024 •

edited

Loading

mikicho Mar 31, 2024 •

edited

Loading

kettanaito Apr 12, 2024 •

edited

Loading