test: skip test-diagnostics-channel-memory-leak.js

[joyeecheung]

test: skip test-diagnostics-channel-memory-leak.js

There is currently no reliable way to detect this leak because:

1. We cannot reliably get a reference to the channel from the
   API to detect finalization without creating another strong reference.
2. This test does gc() and then checks memory usage - however the
   use of gc() disables code aging which can actually lead to increased
   memory usage overall, as it is not intended to be used to lower
   memory usage in the first place.
3. The implementation of diagnostics channels relies on ephemeron gc
   which is inefficient, it's not reliable to use the typical "create
   a lot of objects and see if it crashes" trick to check leaks.

Skip the test for now until we find a way to test it reliably.

To avoid flakiness in the CI, it's better to remove an unreliable
test altogether.

I found this issue from an integration test run in https://chromium-review.googlesource.com/c/v8/v8/+/4962094

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/55120/

[joyeecheung]

Actually, I don't think we can use checkIfCollectable() to check this given how the channels are themselves retained by a WeakMap holding Weakrefs. I am inclined to just remove this test as there is no reliable way to detect this leak

1. Use of gc() in general should be avoided in leak detection tests due to the fact that it disables code aging for the compilation cache (which is why it was failing in the v8 integration test).
2. Because it relies on ephemeron gc which is inefficient, it's not reliable to use the typical "create a lot of objects and see if it crashes" trick.

[joyeecheung]

Updated to just remove this test. @jasnell can you take a look again?

also cc @Qard @theanarkh

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/55300/

[Qard]

Would checking with FinalizationRegistry help? Could make a ton of instances, register each, and then check if any of them ever triggered the finalizer before the process exits. 🤔

[joyeecheung]

Would checking with FinalizationRegistry help? Could make a ton of instances, register each, and then check if any of them ever triggered the finalizer before the process exits. 🤔

That's what I tried with checkIfCollectable() (which does exactly this) and nope it doesn't work, still flaky. Locally the finalizer is triggered after a few instances but in the CI that's a different story. Technically this is also not what FinalizationRegistry guarantees - it's still spec compliant that the finalizer never gets called. I think the lesson learned from #49710 is that FinalizationRegistry with even heap snapshots (which is another inappropriate way to trigger GC) is still unreliable in detecting leaks and leak detection in JS land can only be done as a best-effort. If false positives are inevitable, just don't test it.

[Qard]

I mean, diagnostics_channel itself is using FinalizationRegistry for the cleanup this is supposed to be testing, so if FinalizationRegistry is unreliable isn't that a problem? 🤔

Or is this just an issue of the particular behaviour of --expose_gc? Would there be a way to trigger a deeper GC at the C++ level that we could expose for the tests?

[joyeecheung]

so if FinalizationRegistry is unreliable isn't that a problem? 🤔

Yes, which is why FinalizationRegistry should be avoided if possible https://github.com/tc39/proposal-weakrefs#a-note-of-caution - it seems the diagnostics channel implementation is doing something the proposal specifically advise against, though that's out of the scope of this PR.

Or is this just an issue of the particular behaviour of --expose_gc? Would there be a way to trigger a deeper GC at the C++ level that we could expose for the tests?

--expose_gc (or specifically the forced GC it allows) just alters the GC mechanism slightly in that it disables code aging (so compiled code is ever going to be GC'ed, which lead to the failure of the original test once there's more (unrelated) compiled code in the heap). V8 intentionally does not provide a public API to "trigger a deeper GC" (not one that’s guaranteed to clear specific objects that you think should be cleared anyway) because GC is supposed to be internal to the JavaScript engine implementation. I think any test that rely on "a deeper GC" to clear specific objects especially to deflake is not really worth the maintenance cost. If the test doesn't flake, it's okay to keep them, but if they flake, it's better to not do an unreliable flaky test.

[joyeecheung]

If there are no more concerns about removing the test by Wednesday UTC, I am going to land this change to unblock https://chromium-review.googlesource.com/c/v8/v8/+/4962094 - if it's possible to invent a leak detection test that does not flake, it can always be added later, but I think the current test is just flaky in nature, so it's better to remove it for now.

[Qard]

Approving to unblock this, but I'd prefer to avoid deleting the test if we can. Would adding the test to a test/**.status file as flaky work rather than deleting it? I don't know enough about how the flaky tagging works to know if that would get things passing while still keeping the test around to unmark in the future if we fix it.

As for diagnostics_channel using FinalizationRegistry, it's not ideal that it might not clean it up, but it was basically a solution to dynamically generated channels (which are discouraged) would otherwise stick around forever. It's an attempt to prevent users from shooting themselves in the foot, but perhaps not 100% successfully. If you've got any ideas on how to do it more reliably I would love to hear them. I think it's probably acceptable as it is though as it's hopefully at least better than definitely leaking. As it is, it just may leak an empty WeakReference instance. 😅

[joyeecheung]

I don't know enough about how the flaky tagging works to know if that would get things passing while still keeping the test around to unmark in the future if we fix it.

I think in this case having a flaky test is probably not the best solution, because this would be intentionally making the CI more orange. Maybe we can mark it as SKIP directly if we really want to keep the test?

[Qard]

SKIP would be fine too if that gets us to green without throwing out the context that we were testing for this at some point and should probably be finding a better way to test it at some point in the future. 😅

Otherwise, I'm okay with removing the test entirely if there's not a good way to keep it around without it interfering with our CI. Flaky tests are definitely bad tests. 😬

[nodejs-github-bot]

Landed in 94156e3

[joyeecheung]

An interesting idea came up to me about how to test this somewhat more reliably using a (currently somewhat internal) V8 API (v8::debug::QueryObjects) and the CI looks happy with it https://ci.nodejs.org/job/node-test-commit/66307/ - I'll see if it's possible to upstream my V8 patches to expose this to the embedders, some restrictions may need to be added to prevent the embedders from using this too liberally

[Qard]

I'm all for more reliable leak checking. 🙂

[joyeecheung]

WIP upstream CL in https://chromium-review.googlesource.com/c/v8/v8/+/5006373