macOS nightly deadline/liveliness failed tests

[clalancette]

Last night two of the deadline/liveliness tests failed: https://ci.ros2.org/view/nightly/job/nightly_osx_release/1261/#showFailuresLink. Part of the output from the deadline failure looks like:

/Users/osrf/jenkins-agent/workspace/nightly_osx_release/ws/src/ros2/rcl/rcl/test/rcl/test_events.cpp:464
Expected equality of these values:
  wait_res
    Which is: 2
  0

/Users/osrf/jenkins-agent/workspace/nightly_osx_release/ws/src/ros2/rcl/rcl/test/rcl/test_events.cpp:467
Value of: msg_persist_ready
  Actual: false
Expected: true

/Users/osrf/jenkins-agent/workspace/nightly_osx_release/ws/src/ros2/rcl/rcl/test/rcl/test_events.cpp:486
Expected equality of these values:
  requested_deadline_status.total_count
    Which is: 4
  1

It seems these tests were enabled by ros2/rcl#433 ; @mm318, can you take a look?

[clalancette]

This one was still a problem over the weekend: https://ci.ros2.org/view/nightly/job/nightly_osx_release/1264/#showFailuresLink

[clalancette]

I also saw a failure on aarch64 repeated, though the signature is somewhat different: https://ci.ros2.org/view/nightly/job/nightly_linux-aarch64_repeated/789/#showFailuresLink

[emersonknapp]

Note - I am currently working on this, if we could change the assignee.

Confirmed very reproducible. Should we open this up as an issue on https://github.com/ros2/rcl

System

OSX High Sierra
RMW_IMPLEMENTATION=rmw_connext_cpp

Observations so far

The two following tests inconsistently fail:
test_pubsub_deadline_missed
test_pubsub_liveliness_kill_pub

When running a single test alone repeatedly (via --gtest_filter to select the single test) I have not yet seen either of those tests fail. Each of them takes about 5s to complete on each run and succeeds well past 10 re-runs. However, when running even two of the tests via the same command (e.g. build/rcl/test/test_events__rmw_connext_cpp --gtest_filter="TestEventFixture__rmw_connext_cpp.test_pubsub_*", the tests fail via timeouts on most runs.

Continuing to dig.

[mm318]

I was suggesting that you should try running with --gtest_shuffle and see that the first test case to run will pass and the remaining will fail. It doesn't have to do with particular test cases failing.

[emersonknapp]

It's specifically those two tests - the others are ok, but you're right that when running the two of them that the first passes and the second fails.

EDIT: scratch that,
test_pubsub_no_deadline_missed also is in the set. Only test_unsupported_liveliness passes consistently.

[mm318]

test_unsupported_liveliness is unlike the other tests in that it doesn't involve publishing a message at all; it doesn't do any more than initializing a publisher/subscription.

[emersonknapp]

Noted. Trying to figure out if we're maybe not cleaning up properly in the test fixture, since the first one is always fine.