investigate flaky test/node-api/test_worker_terminate_finalization/test.js #36868
Labels
flaky-test
Issues and PRs related to the tests with unstable failures on the CI.
Comments
aduh95
added
the
flaky-test
|
|
@nodejs/n-api |
This was referenced Jan 13, 2021
|
Does not look like the test has changed in a few months:
|
|
Don't see any recent PRs that look suspect either:
|
|
The log seems to point to this part: Line 367 in 03fd363 Perhaps what |
|
@gabrielschulhof can you think of any recent changes that could be related? |
|
@RaisinTen it is worth seeing what data.GetParameter() looks like. First step would be to recreate locally. I wanted to find some time to do that today but did not get to it. Have you run/recreated on your own machine? |
|
@mhdawson it is indeed. I didn't get time to test this locally either. I'll try it out now. |
|
@mhdawson just ended up running the test a 1000 times and still couldn't replicate the error. :/ |
|
@RaisinTen No recreate for me in 500 runs on RHEL 8 either. |
|
Valgrind with default options does not report anything of interest: [midawson@midawson node]$ valgrind ./node test/node-api/test_worker_terminate_finalization/test.js
==2142587== Memcheck, a memory error detector
==2142587== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==2142587== Using Valgrind-3.16.0 and LibVEX; rerun with -h for copyright info
==2142587== Command: ./node test/node-api/test_worker_terminate_finalization/test.js
==2142587==
==2142587==
==2142587== HEAP SUMMARY:
==2142587== in use at exit: 6,508 bytes in 24 blocks
==2142587== total heap usage: 53,127 allocs, 53,103 frees, 47,206,475 bytes allocated
==2142587==
==2142587== LEAK SUMMARY:
==2142587== definitely lost: 0 bytes in 0 blocks
==2142587== indirectly lost: 0 bytes in 0 blocks
==2142587== possibly lost: 304 bytes in 1 blocks
==2142587== still reachable: 6,204 bytes in 23 blocks
==2142587== suppressed: 0 bytes in 0 blocks
==2142587== Rerun with --leak-check=full to see details of leaked memory
==2142587==
==2142587== For lists of detected and suppressed errors, rerun with: -s
==2142587== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
[midawson@midawson node]$
|
|
Will try re-building with the address sanitizer enabled as it seems some of the failures were with asan enabled and that was what reported the errors |
|
While rebuilding looked at 5-6 of the PRs from today, no recreates in the asan github action runs. |
|
I've been trying to recompile with asan enabled using I've been running |
|
@mhdawson Can you please share the log of the compilation failure? I'm guessing it failed because asan builds consume a lot of memory. Wouldn't Lines 131 to 133 in e3e054d CONFIG_FLAGS would be empty and so:Lines 529 to 531 in e3e054d would basically run: $(PYTHON) ./configure --verbose
make |
|
@RaisinTen the ASAN workflow sets node/.github/workflows/test-asan.yml Line 28 in e3e054d |
|
Right I just did ./configure --verbose --enable-asan
make testbut if fails with these kinds of errors: `_ZZN4node7ReallocIcEEPT_S2_mE19__PRETTY_FUNCTION__' referenced in section `.data' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_hash.o: defined in discarded section `.rodata._ZZN4node7ReallocIcEEPT_S2_mE19__PRETTY_FUNCTION__[_ZZN4node7ReallocIcEEPT_S2_mE4args]' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_hash.o
`_ZZN4node16MaybeStackBufferIcLm1024EE25AllocateSufficientStorageEmE19__PRETTY_FUNCTION__' referenced in section `.data' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_hash.o: defined in discarded section `.rodata._ZZN4node16MaybeStackBufferIcLm1024EE25AllocateSufficientStorageEmE19__PRETTY_FUNCTION__[_ZZN4node16MaybeStackBufferIcLm1024EE25AllocateSufficientStorageEmE4args]' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_hash.o
`_ZZN4node11Environment29DecreaseWaitingRequestCounterEvE19__PRETTY_FUNCTION__' referenced in section `.data' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_hash.o: defined in discarded section `.rodata._ZZN4node11Environment29DecreaseWaitingRequestCounterEvE19__PRETTY_FUNCTION__[_ZZN4node11Environment29DecreaseWaitingRequestCounterEvE4args]' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_hash.o
`_ZZN4node14ThreadPoolWork12ScheduleWorkEvE19__PRETTY_FUNCTION__' referenced in section `.data' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_hash.o: defined in discarded section `.rodata._ZZN4node14ThreadPoolWork12ScheduleWorkEvE19__PRETTY_FUNCTION__[_ZZN4node14ThreadPoolWork12ScheduleWorkEvE4args]' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_hash.o
`_ZZN4node14ThreadPoolWorkC4EPNS_11EnvironmentEE19__PRETTY_FUNCTION__' referenced in section `.data' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_hash.o: defined in discarded section `.rodata._ZZN4node14ThreadPoolWorkC4EPNS_11EnvironmentEE19__PRETTY_FUNCTION__[_ZZN4node14ThreadPoolWorkC4EPNS_11EnvironmentEE4args]' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_hash.o
`_ZZN4node18MemoryRetainerNodeC4EPNS_13MemoryTrackerEPKNS_14MemoryRetainerEE19__PRETTY_FUNCTION__' referenced in section `.data' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_hash.o: defined in discarded section `.rodata._ZZN4node18MemoryRetainerNodeC4EPNS_13MemoryTrackerEPKNS_14MemoryRetainerEE19__PRETTY_FUNCTION__[_ZZN4node18MemoryRetainerNodeC4EPNS_13MemoryTrackerEPKNS_14MemoryRetainerEE4args]' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_hash.o
`_ZZN4node10BaseObjectD4EvE19__PRETTY_FUNCTION__' referenced in section `.data' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_hash.o: defined in discarded section `.rodata._ZZN4node10BaseObjectD4EvE19__PRETTY_FUNCTION__[_ZZN4node10BaseObjectD4EvE4args]' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_hash.o
`_ZZZN4node10BaseObject8MakeWeakEvENKUlRKN2v816WeakCallbackInfoIS0_EEE_clES5_E19__PRETTY_FUNCTION__' referenced in section `.data' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_hash.o: defined in discarded section `.rodata._ZZZN4node10BaseObject8MakeWeakEvENKUlRKN2v816WeakCallbackInfoIS0_EEE_clES5_E19__PRETTY_FUNCTION__[_ZZZN4node10BaseObject8MakeWeakEvENKUlRKN2v816WeakCallbackInfoIS0_EEE_clES5_E4args]' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_hash.o
`_ZZN4node11Environment14AddCleanupHookEPFvPvES1_E19__PRETTY_FUNCTION__' referenced in section `.data' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_hash.o: defined in discarded section `.rodata._ZZN4node11Environment14AddCleanupHookEPFvPvES1_E19__PRETTY_FUNCTION__[_ZZN4node11Environment14AddCleanupHookEPFvPvES1_E4args]' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_hash.o
`_ZZN4node11Environment29DecreaseWaitingRequestCounterEvE19__PRETTY_FUNCTION__' referenced in section `.data' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_keygen.o: defined in discarded section `.rodata._ZZN4node11Environment29DecreaseWaitingRequestCounterEvE19__PRETTY_FUNCTION__[_ZZN4node11Environment29DecreaseWaitingRequestCounterEvE4args]' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_keygen.o
`_ZZN4node14ThreadPoolWork12ScheduleWorkEvE19__PRETTY_FUNCTION__' referenced in section `.data' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_keygen.o: defined in discarded section `.rodata._ZZN4node14ThreadPoolWork12ScheduleWorkEvE19__PRETTY_FUNCTION__[_ZZN4node14ThreadPoolWork12ScheduleWorkEvE4args]' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_keygen.o
`_ZZZN4node10BaseObject8MakeWeakEvENKUlRKN2v816WeakCallbackInfoIS0_EEE_clES5_E19__PRETTY_FUNCTION__' referenced in section `.data' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_keygen.o: defined in discarded section `.rodata._ZZZN4node10BaseObject8MakeWeakEvENKUlRKN2v816WeakCallbackInfoIS0_EEE_clES5_E19__PRETTY_FUNCTION__[_ZZZN4node10BaseObject8MakeWeakEvENKUlRKN2v816WeakCallbackInfoIS0_EEE_clES5_E4args]' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_keygen.o
`_ZZN4node14ThreadPoolWorkC4EPNS_11EnvironmentEE19__PRETTY_FUNCTION__' referenced in section `.data' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_keygen.o: defined in discarded section `.rodata._ZZN4node14ThreadPoolWorkC4EPNS_11EnvironmentEE19__PRETTY_FUNCTION__[_ZZN4node14ThreadPoolWorkC4EPNS_11EnvironmentEE4args]' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_keygen.o
`_ZZN4node18MemoryRetainerNodeC4EPNS_13MemoryTrackerEPKNS_14MemoryRetainerEE19__PRETTY_FUNCTION__' referenced in section `.data' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_keygen.o: defined in discarded section `.rodata._ZZN4node18MemoryRetainerNodeC4EPNS_13MemoryTrackerEPKNS_14MemoryRetainerEE19__PRETTY_FUNCTION__[_ZZN4node18MemoryRetainerNodeC4EPNS_13MemoryTrackerEPKNS_14MemoryRetainerEE4args]' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_keygen.o
`_ZZN4node6crypto13MallocOpenSSLIcEEPT_mE19__PRETTY_FUNCTION__' referenced in section `.data' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_keygen.o: defined in discarded section `.rodata._ZZN4node6crypto13MallocOpenSSLIcEEPT_mE19__PRETTY_FUNCTION__[_ZZN4node6crypto13MallocOpenSSLIcEEPT_mE4args]' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_keygen.o
`_ZZN4node11SPrintFImplB5cxx11EPKcE19__PRETTY_FUNCTION__' referenced in section `.data' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_keygen.o: defined in discarded section `.rodata._ZZN4node11SPrintFImplB5cxx11EPKcE19__PRETTY_FUNCTION__[_ZZN4node11SPrintFImplB5cxx11EPKcE4args]' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_keygen.o
`_ZZN4node6crypto13MallocOpenSSLIcEEPT_mE19__PRETTY_FUNCTION__' referenced in section `.data' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_scrypt.o: defined in discarded section `.rodata._ZZN4node6crypto13MallocOpenSSLIcEEPT_mE19__PRETTY_FUNCTION__[_ZZN4node6crypto13MallocOpenSSLIcEEPT_mE4args]' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_scrypt.o
`_ZZN4node6crypto25ArrayBufferOrViewContentsIcEC4EN2v85LocalINS3_5ValueEEEE19__PRETTY_FUNCTION__' referenced in section `.data' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_scrypt.o: defined in discarded section `.rodata._ZZN4node6crypto25ArrayBufferOrViewContentsIcEC4EN2v85LocalINS3_5ValueEEEE19__PRETTY_FUNCTION__[_ZZN4node6crypto25ArrayBufferOrViewContentsIcEC4EN2v85LocalINS3_5ValueEEEE4args]' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_scrypt.o
`_ZZN4node18MemoryRetainerNodeC4EPNS_13MemoryTrackerEPKNS_14MemoryRetainerEE19__PRETTY_FUNCTION__' referenced in section `.data' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_x509.o: defined in discarded section `.rodata._ZZN4node18MemoryRetainerNodeC4EPNS_13MemoryTrackerEPKNS_14MemoryRetainerEE19__PRETTY_FUNCTION__[_ZZN4node18MemoryRetainerNodeC4EPNS_13MemoryTrackerEPKNS_14MemoryRetainerEE4args]' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_x509.o
`_ZZN4node10BaseObjectD4EvE19__PRETTY_FUNCTION__' referenced in section `.data' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_x509.o: defined in discarded section `.rodata._ZZN4node10BaseObjectD4EvE19__PRETTY_FUNCTION__[_ZZN4node10BaseObjectD4EvE4args]' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_x509.o
`_ZZZN4node10BaseObject8MakeWeakEvENKUlRKN2v816WeakCallbackInfoIS0_EEE_clES5_E19__PRETTY_FUNCTION__' referenced in section `.data' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_x509.o: defined in discarded section `.rodata._ZZZN4node10BaseObject8MakeWeakEvENKUlRKN2v816WeakCallbackInfoIS0_EEE_clES5_E19__PRETTY_FUNCTION__[_ZZZN4node10BaseObject8MakeWeakEvENKUlRKN2v816WeakCallbackInfoIS0_EEE_clES5_E4args]' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_x509.o
`_ZZN4node11Environment14AddCleanupHookEPFvPvES1_E19__PRETTY_FUNCTION__' referenced in section `.data' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_x509.o: defined in discarded section `.rodata._ZZN4node11Environment14AddCleanupHookEPFvPvES1_E19__PRETTY_FUNCTION__[_ZZN4node11Environment14AddCleanupHookEPFvPvES1_E4args]' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_x509.o
`_ZZN4node6crypto13GetInfoStringILi177EEEN2v810MaybeLocalINS2_5ValueEEEPNS_11EnvironmentERKSt10unique_ptrI6bio_stNS_15FunctionDeleterIS9_XadL_Z12BIO_free_allEEEEEP7x509_stE19__PRETTY_FUNCTION__' referenced in section `.data' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_x509.o: defined in discarded section `.rodata._ZZN4node6crypto13GetInfoStringILi177EEEN2v810MaybeLocalINS2_5ValueEEEPNS_11EnvironmentERKSt10unique_ptrI6bio_stNS_15FunctionDeleterIS9_XadL_Z12BIO_free_allEEEEEP7x509_stE19__PRETTY_FUNCTION__[_ZZN4node6crypto13GetInfoStringILi177EEEN2v810MaybeLocalINS2_5ValueEEEPNS_11EnvironmentERKSt10unique_ptrI6bio_stNS_15FunctionDeleterIS9_XadL_Z12BIO_free_allEEEEEP7x509_stE4args]' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_x509.o
`_ZZN4node6crypto13GetInfoStringILi85EEEN2v810MaybeLocalINS2_5ValueEEEPNS_11EnvironmentERKSt10unique_ptrI6bio_stNS_15FunctionDeleterIS9_XadL_Z12BIO_free_allEEEEEP7x509_stE19__PRETTY_FUNCTION__' referenced in section `.data' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_x509.o: defined in discarded section `.rodata._ZZN4node6crypto13GetInfoStringILi85EEEN2v810MaybeLocalINS2_5ValueEEEPNS_11EnvironmentERKSt10unique_ptrI6bio_stNS_15FunctionDeleterIS9_XadL_Z12BIO_free_allEEEEEP7x509_stE19__PRETTY_FUNCTION__[_ZZN4node6crypto13GetInfoStringILi85EEEN2v810MaybeLocalINS2_5ValueEEEPNS_11EnvironmentERKSt10unique_ptrI6bio_stNS_15FunctionDeleterIS9_XadL_Z12BIO_free_allEEEEEP7x509_stE4args]' of /home/midawson/io.js/out/Release/obj.target/libnode/src/crypto/crypto_x509.o
collect2: error: ld returned 1 exit statusI've tried on a few machines, but I'm guessing I must be missing something that needs to be installed, have not had time to look at the error messages to figure out what yet though. |
|
I just can't get it to work with asan. I even tried on ubuntu 18.04 which is what I think it runs on in the CI. The error I'm getting there is: |
|
Looking at the history of the github action I think the difference is that it uses clang instead of gcc: |
|
hmm, that did not seem to fix it either :( |
|
Seems like the latest issue is that my latest runs are docker and that it needs privileges that are not available by default. |
|
ok got it to run, no recreate on first run under asan. |
|
Ok 1150 local runs of test_worker_terminate_finalization with asan enabled and nothing reported by asan and no recreates. Since we've not been able to recreate locally and we have not seen it recreate in any of the CI runs since I think we should close and if it re-occurs we can figure out if there are any other ways to recreate. |
|
I also tried to reproduce this crash locally with both gcc and clang on current master (ad3ebed) and was unable to reproduce this. I believe that #37303 fixes a different issue, as I explained in #37303 (comment). So, until we find a reliable repro for this issue, or until flakiness of this test is once again reported, IMO we should close this issue. Feel free to re-open though, if you believe it should be open. |
|
@gabrielschulhof Thank you for looking into this. I had reopened this since |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Refs: #34731
The text was updated successfully, but these errors were encountered: