[FlexCounter] Fix shutdown syncd hanging problem #1006
Conversation
* If the flexCounter try to end the thread while the thread is collecting counter, it will miss the m_cvSleep.notify_all() After collecting counter, it will still execute m_cvSleep.wait_for() and be idle for about 10 seconds. * Check if the m_runFlexCounterThread is disabled before executing the wait_for() command
|
|
||
| m_cvSleep.wait_for(lk, std::chrono::milliseconds(correction)); | ||
| // If m_runFlexCounterThread is already disabled, do not wait for signal | ||
| if (m_runFlexCounterThread) |
There was a problem hiding this comment.
how this was hanging ? if m_runFlexCounterThread is already false, then wait_for will wait for correction time and in next loop iteration it will break since m_runFlexCounterThread is false on line 2360
so where was hanging problem? was that deadlock? is m_mtxSleep used somewhere else?
There was a problem hiding this comment.
also this does not solve problem completely, since ther can be race condition, and m_runFlexCounters can be TRUE on line 2384 but it can be set to FALSE at line 2386 by other thread, so swill wait_for will be executed
There was a problem hiding this comment.
It won't hang forever but about 10 seconds for some kinds of counter thread. And the 10 seconds is also a threshold of the test. The root cause just like your second comment. The m_runFlexCounterThread turns from TRUE to FALSE before the thread run to line 2385.
This is not a solution to solve the race condition but just a simple trick to solve 99% hanging 10 seconds situation.
There was a problem hiding this comment.
i would prefer to solve this completely to not get back to this
What I did
Add a check before the FlexCounter thread execute the wait_for() funciton.
Why I did it
I encountered syncd shutdown overtime issue during running the test, sonic-mgmt/ansible/roles/test/tasks/restart_syncd.yml
I found it is due to it stuck by FlexCounter during removeAllCounters()
How I verified it
After modification, run the test for over 50 times
Additional information
The following is the syslog when I encountered the failure test.