-
Notifications
You must be signed in to change notification settings - Fork 9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
regression: js<->go integration test hangs with version 6.0.1 vs. 5.0.4 #71
Comments
NOTE: The following issues must be addressed before this is merged: libp2p/interface is currently held at version 0.1.2: - Passing a logger to MuxerInit is now required (should be optional?): libp2p/js-libp2p#2275 @chainsafe/libp2p-yamux is currently held at version 5.0.4: - Passing a logger to yamux()({logger:...}) is now required (should be optional?): ChainSafe/js-libp2p-yamux#69 - The integration test for rpcstream fails with 6.x vs. 5.x: ChainSafe/js-libp2p-yamux#71 and https://github.com/aperturerobotics/starpc/actions/runs/7055030516/job/19204891175 See: #120 --- Update dependencies and fix various breaking changes introduced by those dependency updates: - Change type of Conn to Duplex<Uint8Array | Uint8ArrayList> - Pass a logger stub to yamux as it's now required. The logger stub should not be necessary once these are fixed: - ChainSafe/js-libp2p-yamux#69 - libp2p/js-libp2p#2275 - ChainSafe/js-libp2p-yamux#70 Signed-off-by: Christian Stewart <christian@aperture.us>
NOTE: The following issues must be addressed before this is merged: libp2p/interface is currently held at version 0.1.2: - Passing a logger to MuxerInit is now required (should be optional?): libp2p/js-libp2p#2275 @chainsafe/libp2p-yamux is currently held at version 5.0.4: - Passing a logger to yamux()({logger:...}) is now required (should be optional?): ChainSafe/js-libp2p-yamux#69 - The integration test for rpcstream fails with 6.x vs. 5.x: ChainSafe/js-libp2p-yamux#71 and https://github.com/aperturerobotics/starpc/actions/runs/7055030516/job/19204891175 See: #120 --- Update dependencies and fix various breaking changes introduced by those dependency updates: - Change type of Conn to Duplex<Uint8Array | Uint8ArrayList> - Pass a logger stub to yamux as it's now required. The logger stub should not be necessary once these are fixed: - ChainSafe/js-libp2p-yamux#69 - libp2p/js-libp2p#2275 - ChainSafe/js-libp2p-yamux#70 Signed-off-by: Christian Stewart <christian@aperture.us>
See: - #120 - libp2p/js-libp2p#2275 - ChainSafe/js-libp2p-yamux#71 Signed-off-by: Christian Stewart <christian@aperture.us>
NOTE: The following issues must be addressed before this is merged: libp2p/interface is currently held at version 0.1.2: - Passing a logger to MuxerInit is now required (should be optional?): libp2p/js-libp2p#2275 @chainsafe/libp2p-yamux is currently held at version 5.0.4: - Passing a logger to yamux()({logger:...}) is now required (should be optional?): ChainSafe/js-libp2p-yamux#69 - The integration test for rpcstream fails with 6.x vs. 5.x: ChainSafe/js-libp2p-yamux#71 and https://github.com/aperturerobotics/starpc/actions/runs/7055030516/job/19204891175 See: #120 --- Update dependencies and fix various breaking changes introduced by those dependency updates: - Change type of Conn to Duplex<Uint8Array | Uint8ArrayList> - Pass a logger stub to yamux as it's now required. The logger stub should not be necessary once these are fixed: - ChainSafe/js-libp2p-yamux#69 - libp2p/js-libp2p#2275 - ChainSafe/js-libp2p-yamux#70 Signed-off-by: Christian Stewart <christian@aperture.us>
Do you have any more details on this? |
@wemeetagain Sure, can reproduce the issue like so:
You can see that the updated version hangs. If I rollback yamux to 5.x, it works. |
Haven't looked much deeper into this, but I at least enabled yamux logs to see what's happening on the js side. diff --git a/srpc/log.ts b/srpc/log.ts
index bbfe6ab..2af47bb 100644
--- a/srpc/log.ts
+++ b/srpc/log.ts
@@ -1,4 +1,5 @@
import type { ComponentLogger, Logger } from '@libp2p/interface'
+import { defaultLogger } from '@libp2p/logger'
// https://github.com/libp2p/js-libp2p/issues/2276
// https://github.com/libp2p/js-libp2p/blob/bca8d6e689b47d85dda74082ed72e671139391de/packages/logger/src/index.ts#L86
@@ -21,5 +22,6 @@ export function createDisabledLogger(namespace: string): Logger {
}
export function createDisabledComponentLogger(): ComponentLogger {
+ return defaultLogger()
return { forComponent: createDisabledLogger }
}
So js is sending a new stream open message (type 1, flag 1), sending data (type 0, flag 0), some ping back and forth (type 2), we receive an acknowledgement of the stream being opened (type 1, flag 2), then it hangs (right before we receiving data back). |
@wemeetagain Thanks for taking a look at this. I updated the |
I can replicate this locally. The change in What I've noticed is if I patch the muxer diff --git a/src/muxer.ts b/src/muxer.ts
index df4e792..502dc7e 100644
--- a/src/muxer.ts
+++ b/src/muxer.ts
@@ -561,9 +561,8 @@ export class YamuxMuxer implements StreamMuxer {
if (data === undefined) {
throw new CodeError('invalid frame', ERR_INVALID_FRAME)
}
- this.source.push(
- new Uint8ArrayList(encodeHeader(header), data)
- )
+ this.source.push(encodeHeader(header))
+ this.source.push(data.subarray())
} else {
this.source.push(encodeHeader(header))
} Can you point to the bit of starpc that consumes the output from the muxer? I think it may not be handling |
Good catch. I'll have a look |
@achingbrain Found the issue: now yamux is emitting Uint8ArrayList objects, and I pass those to The fix was to simply add a step to the pipe to merge together all the Uint8ArrayList elements before passing to it-ws: https://github.com/aperturerobotics/starpc/pull/121/files#diff-ebbf8f0716a15b0ffad17f77ecd8ee9566f1941b8a921087552cd13ce92e7e6eR24 Thanks very much for the assistance here 👍🏽 |
Good stuff, glad you got it sorted. One other thing to consider is when turning async function * copyTransform (source) {
for await (const list of source) {
yield list.subarray()
}
} or you can just yield the backing async function * noCopyTransform (source) {
for await (const list of source) {
yield * list
}
} The advantage of the second method is you use a lot less memory, the disadvantage is you write lots of small bits of data into the transport which can incur some overhead so it's worth testing which is more efficient for your application, but generally not copying the data is better. * This only actually copies if the |
The update from 5.0.4 to 6.0.1 breaks my integration tests between Go and Js yamux:
https://github.com/aperturerobotics/starpc/actions/runs/7055030516/job/19204891175
It works in most cases but there's a specific test which hangs under 6.0.1 and works under 5.0.4.
I haven't been able to find the root cause of the issue yet, but definitely have confirmed the regression is in 6.0.1 vs 5.0.4.
The text was updated successfully, but these errors were encountered: