Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky CarbonRouterClient.basicUsageSameThreadClient test #458

Conversation

mszabo-wikia
Copy link
Contributor

CarbonRouterClient.basicUsageSameThreadClient currently does not wait for code running on the proxy EVB to complete, which causes flakiness and errors like the following:

*** Aborted at 1734013374 (Unix time, try 'date -d @1734013374') ***
*** Signal 11 (SIGSEGV) (0x0) received by PID 72130 (pthread TID 0x7f4b45a096c0) (linux TID 72132) (code: 128), stack trace: ***
    @ 00000000007f0586 folly::symbolizer::(anonymous namespace)::signalHandler(int, siginfo_t*, void*)
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/debugging/symbolizer/SignalHandler.cpp:453
    @ 0000000000019dcf (unknown)
    @ 000000000078cdb9 folly::TimeoutManager::clearCobTimeouts()
                       /usr/include/boost/intrusive/detail/list_node.hpp:60
                       -> /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/TimeoutManager.cpp
    @ 000000000078df70 folly::VirtualEventBase::destroyImpl()
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/VirtualEventBase.cpp:38
    @ 00000000007739e6 bool folly::AtomicNotificationQueue<folly::Function<void ()> >::drive<folly::EventBase::FuncRunner&>(folly::EventBase::FuncRunner&)
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/Function.h:370
                       -> /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventBase.cpp
    @ 0000000000775183 folly::EventBaseAtomicNotificationQueue<folly::Function<void ()>, folly::EventBase::FuncRunner>::execute()
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventBaseAtomicNotificationQueue-inl.h:270
                       -> /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventBase.cpp
    @ 000000000077521c non-virtual thunk to folly::EventBaseAtomicNotificationQueue<folly::Function<void ()>, folly::EventBase::FuncRunner>::handlerReady(unsigned short)
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventBaseAtomicNotificationQueue-inl.h:279
                       -> /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventBase.cpp
    @ 000000000077a783 folly::EventHandler::libeventCallback(int, short, void*)
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventHandler.cpp:159
    @ 00000000000247f9 (unknown)
    @ 000000000002642e event_base_loop
    @ 000000000076e331 folly::EventBase::loopMain(int, folly::EventBase::LoopOptions)
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventBase.cpp:99
    @ 000000000076e74d folly::EventBase::loopBody(int, folly::EventBase::LoopOptions)
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventBase.cpp:516
    @ 000000000076e7e7 folly::EventBase::loop()
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventBase.cpp:477
    @ 0000000000771336 folly::EventBase::loopForever()
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventBase.cpp:784
    @ 00000000007221eb folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/executors/IOThreadPoolExecutor.cpp:241
    @ 000000000072c557 void folly::detail::function::call_<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)>, true, false, void>(, folly::detail::function::Data&)
                       /usr/include/c++/14/bits/invoke.h:74
                       -> /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/executors/ThreadPoolExecutor.cpp
    @ 000000000004b523 (unknown)
    @ 0000000000070cd6 start_thread
    @ 00000000000f4c8b __clone3

As a fix, add explicit synchronization to ensure the code scheduled on the EVB completes first before attempting to terminate the EVB.

CarbonRouterClient.basicUsageSameThreadClient currently does not wait
for code running on the proxy EVB to complete, which causes flakiness
and errors like the following:

```
*** Aborted at 1734013374 (Unix time, try 'date -d @1734013374') ***
*** Signal 11 (SIGSEGV) (0x0) received by PID 72130 (pthread TID 0x7f4b45a096c0) (linux TID 72132) (code: 128), stack trace: ***
    @ 00000000007f0586 folly::symbolizer::(anonymous namespace)::signalHandler(int, siginfo_t*, void*)
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/debugging/symbolizer/SignalHandler.cpp:453
    @ 0000000000019dcf (unknown)
    @ 000000000078cdb9 folly::TimeoutManager::clearCobTimeouts()
                       /usr/include/boost/intrusive/detail/list_node.hpp:60
                       -> /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/TimeoutManager.cpp
    @ 000000000078df70 folly::VirtualEventBase::destroyImpl()
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/VirtualEventBase.cpp:38
    @ 00000000007739e6 bool folly::AtomicNotificationQueue<folly::Function<void ()> >::drive<folly::EventBase::FuncRunner&>(folly::EventBase::FuncRunner&)
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/Function.h:370
                       -> /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventBase.cpp
    @ 0000000000775183 folly::EventBaseAtomicNotificationQueue<folly::Function<void ()>, folly::EventBase::FuncRunner>::execute()
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventBaseAtomicNotificationQueue-inl.h:270
                       -> /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventBase.cpp
    @ 000000000077521c non-virtual thunk to folly::EventBaseAtomicNotificationQueue<folly::Function<void ()>, folly::EventBase::FuncRunner>::handlerReady(unsigned short)
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventBaseAtomicNotificationQueue-inl.h:279
                       -> /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventBase.cpp
    @ 000000000077a783 folly::EventHandler::libeventCallback(int, short, void*)
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventHandler.cpp:159
    @ 00000000000247f9 (unknown)
    @ 000000000002642e event_base_loop
    @ 000000000076e331 folly::EventBase::loopMain(int, folly::EventBase::LoopOptions)
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventBase.cpp:99
    @ 000000000076e74d folly::EventBase::loopBody(int, folly::EventBase::LoopOptions)
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventBase.cpp:516
    @ 000000000076e7e7 folly::EventBase::loop()
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventBase.cpp:477
    @ 0000000000771336 folly::EventBase::loopForever()
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventBase.cpp:784
    @ 00000000007221eb folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/executors/IOThreadPoolExecutor.cpp:241
    @ 000000000072c557 void folly::detail::function::call_<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)>, true, false, void>(, folly::detail::function::Data&)
                       /usr/include/c++/14/bits/invoke.h:74
                       -> /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/executors/ThreadPoolExecutor.cpp
    @ 000000000004b523 (unknown)
    @ 0000000000070cd6 start_thread
    @ 00000000000f4c8b __clone3
```

As a fix, add explicit synchronization to ensure the code scheduled on
the EVB completes first before attempting to terminate the EVB.
@mszabo-wikia mszabo-wikia force-pushed the fix-flaky-carbonrouterclient-test branch from 1650519 to 7cfd7a0 Compare December 19, 2024 09:46
@facebook-github-bot
Copy link
Contributor

@stuclar has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@stuclar merged this pull request in efeaca4.

facebook-github-bot pushed a commit to facebook/hhvm that referenced this pull request Dec 20, 2024
Summary:
CarbonRouterClient.basicUsageSameThreadClient currently does not wait for code running on the proxy EVB to complete, which causes flakiness and errors like the following:

```
*** Aborted at 1734013374 (Unix time, try 'date -d 1734013374') ***
*** Signal 11 (SIGSEGV) (0x0) received by PID 72130 (pthread TID 0x7f4b45a096c0) (linux TID 72132) (code: 128), stack trace: ***
    @ 00000000007f0586 folly::symbolizer::(anonymous namespace)::signalHandler(int, siginfo_t*, void*)
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/debugging/symbolizer/SignalHandler.cpp:453
    @ 0000000000019dcf (unknown)
    @ 000000000078cdb9 folly::TimeoutManager::clearCobTimeouts()
                       /usr/include/boost/intrusive/detail/list_node.hpp:60
                       -> /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/TimeoutManager.cpp
    @ 000000000078df70 folly::VirtualEventBase::destroyImpl()
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/VirtualEventBase.cpp:38
    @ 00000000007739e6 bool folly::AtomicNotificationQueue<folly::Function<void ()> >::drive<folly::EventBase::FuncRunner&>(folly::EventBase::FuncRunner&)
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/Function.h:370
                       -> /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventBase.cpp
    @ 0000000000775183 folly::EventBaseAtomicNotificationQueue<folly::Function<void ()>, folly::EventBase::FuncRunner>::execute()
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventBaseAtomicNotificationQueue-inl.h:270
                       -> /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventBase.cpp
    @ 000000000077521c non-virtual thunk to folly::EventBaseAtomicNotificationQueue<folly::Function<void ()>, folly::EventBase::FuncRunner>::handlerReady(unsigned short)
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventBaseAtomicNotificationQueue-inl.h:279
                       -> /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventBase.cpp
    @ 000000000077a783 folly::EventHandler::libeventCallback(int, short, void*)
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventHandler.cpp:159
    @ 00000000000247f9 (unknown)
    @ 000000000002642e event_base_loop
    @ 000000000076e331 folly::EventBase::loopMain(int, folly::EventBase::LoopOptions)
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventBase.cpp:99
    @ 000000000076e74d folly::EventBase::loopBody(int, folly::EventBase::LoopOptions)
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventBase.cpp:516
    @ 000000000076e7e7 folly::EventBase::loop()
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventBase.cpp:477
    @ 0000000000771336 folly::EventBase::loopForever()
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/io/async/EventBase.cpp:784
    @ 00000000007221eb folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)
                       /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/executors/IOThreadPoolExecutor.cpp:241
    @ 000000000072c557 void folly::detail::function::call_<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)>, true, false, void>(, folly::detail::function::Data&)
                       /usr/include/c++/14/bits/invoke.h:74
                       -> /home/mszabo/Documents/fbcode-scratch/repos/git.luolix.top-facebook-folly.git/folly/executors/ThreadPoolExecutor.cpp
    @ 000000000004b523 (unknown)
    @ 0000000000070cd6 start_thread
    @ 00000000000f4c8b __clone3
```

As a fix, add explicit synchronization to ensure the code scheduled on the EVB completes first before attempting to terminate the EVB.

X-link: facebook/mcrouter#458

Reviewed By: disylh

Differential Revision: D67521838

Pulled By: stuclar

fbshipit-source-id: 6f90bacea961a04054f31a9926a4fb2f9958a439
@mszabo-wikia
Copy link
Contributor Author

Thanks!

@mszabo-wikia mszabo-wikia deleted the fix-flaky-carbonrouterclient-test branch December 20, 2024 19:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants