Wait robustness #504

rnorth · 2017-11-26T20:32:15Z

We've been noticing some occasional random test failures particularly in the Selenium container test suites, in addition to troubling log messages where the VNC recording container has had to be restarted (within its retry budget of 3), and possible corrupt/cut-off video files (#466). The latter problem looks to be a race condition where Selenium started listening OK, but the VNC server was not always available yet.

So, I've done some refactoring. In brief:

We now allow a list of multiple startup liveness check ports to be defined for a container. This helps, e.g. for the browser containers, and lets us wait for both Selenium and VNC ports to be listening. I think this will help eliminate random flapping tests in this area.
Also, we now have a WaitAllStrategy that lets more than one wait strategy be used. Again for browser containers, we now wait for (a) a log message, and (b) the listening ports to be available.
For cases where we check running state from within the container, I've added one additional command that can identify listening ports.
I've broken out some aspects of the wait strategies/port detection into separate classes and used this to help improve test coverage.

bsideup · 2017-11-26T20:55:02Z

core/src/main/java/org/testcontainers/containers/GenericContainer.java

@@ -337,6 +342,32 @@ protected Integer getLivenessCheckPort() {
        }
    }

+    /**
+     * @return the port on which to check if the container is ready


should be plural after this change

bsideup · 2017-11-26T20:57:08Z

core/src/main/java/org/testcontainers/containers/GenericContainer.java

+    /**
+     * @return the port on which to check if the container is ready
+     */
+    protected List<Integer> getLivenessCheckPorts() {


since the implementation uses Set as a type of collection, why don't we return Set here as well? Would make more sense, also List indicates the order

bsideup · 2017-11-26T20:57:59Z

core/src/main/java/org/testcontainers/containers/GenericContainer.java

-                    .filter(container -> container.getNames()[0].endsWith(linkableContainer.getContainerName()))
-                    .map(container -> new Link(container.getNames()[0], alias))
+            Set<Link> links = dockerClient.listContainersCmd()
+                    .withStatusFilter("running")


Ah yes - forgot to mention this in the summary. It seems that we had a redundant step here. While we're planning to kill off links, it seemed like a good fix to make all the same.

bsideup · 2017-11-26T20:59:31Z

core/src/main/java/org/testcontainers/containers/wait/HostPortWaitStrategy.java

-        if (null == port) {
-            log.debug("Liveness check port of {} is empty. Not waiting.", container.getContainerName());
+        final List<Integer> externalLivenessCheckPorts = getLivenessCheckPorts();
+        if (null == externalLivenessCheckPorts || externalLivenessCheckPorts.isEmpty()) {


since getLivenessCheckPorts returns a collection, it would be nice to mark it as @NotNull to avoid such checks and make the contract more strict (nullable collection doesn't make a lot of sense anyway :) )

bsideup · 2017-11-26T21:00:41Z

...main/java/org/testcontainers/containers/wait/internal/InternalCommandPortListeningCheck.java

+
+    private void tryPort(Integer internalPort) {
+        String[][] commands = {
+                {"/bin/sh", "-c", format("cat /proc/net/tcp | awk '{print $2}' | grep :%x && echo %s", internalPort, SUCCESS_MARKER)},


bsideup · 2017-11-26T21:01:56Z

.../java/org/testcontainers/containers/wait/internal/InternalCommandPortListeningCheckTest.java

+
+    @Before
+    public void setUp() {
+        nginx = new GenericContainer<>("nginx:1.9.4");


why not as a rule? :)

You're right, I thought this avoided a problem but it doesn't!

bsideup · 2017-11-26T21:02:42Z

modules/jdbc/src/main/java/org/testcontainers/containers/JdbcDatabaseContainer.java

@@ -149,9 +149,6 @@ protected void optionallyMapResourceParameterAsVolume(@NotNull String paramName,
        }
    }

-    @Override
-    protected abstract Integer getLivenessCheckPort();


breaks the binary compatibility

This method, with the same signature, is in GenericContainer, so I was actually thinking that overriding it here makes no sense. (TBH I didn't release you could override a concrete method with an abstract one, so this gives me headaches 🤣 ). Happy to withdraw if you can see the problem more clearly than I can though.

rnorth · 2017-11-26T22:37:19Z

Hmm, based on latest tests on Travis this hasn't improved test stability on CI (actually it's worsened, as I removed the 3x retries for starting VNC containers...).

It looks like we have the original issue of the VNC recorder connecting too soon, with a Connection refused error. So, I'm wondering if I need to dig into how the latest Docker on linux works and whether it might be doing the same proxying that led us to do the in-container port checks.

Thinking out loud, maybe we should just do the port checks in-container all the time...

rnorth · 2017-11-26T22:38:24Z

Just to add, I'm running this apparently flaky test locally 100x (on a Mac) to see if there is any flakiness here.

rnorth · 2017-11-27T09:31:28Z

I left a single selenium test looping overnight in two different modes:

Docker for Mac (internal check): 100% success rate
On linux (inside a container with docker socket mounted): ~20% success rate (bear in mind that this is with VNC recorder retries set to zero)

The Travis build on this branch was also not healthy.

So, I suspect that our external check is not good enough, even when run from a linux host. It's not a userland proxy, but I'm wondering if there's still a race (I'm not sure of the mechanics of the iptables routing setup between containers).

The latest commit, 9c716a8, makes a change that seems to be solid: Regardless of environment, always run the internal check, followed by the external check. So far, this works 100% of the time in both the modes I mentioned above.

kiview · 2017-11-29T16:48:04Z

Also keep in mind, that there might be some bugs/race conditions in our LogMessageWaitStrategy:
#455

Did you notice any problems with the LogMessageWaitStrategyfor the Selenium containers?

rnorth · 2017-11-29T17:55:27Z

Good point. Not noticed that issue with selenium but then again I’ve not personally used them on windows. Def worth reconsidering if the log wait adds value here or if it’s too risky while we have that bug. It’s also made me think maybe the ‘wait for all sequentially’ strategy could be used to fix #455 too in some form...

kiview · 2017-11-30T09:10:37Z

We've monitored this behavior on Linux (Ubuntu and Fedora) as well, so it does not seem to be Windows specific.

"wait for all sequentially" is a nice pragmatic solution, a shame that we would overshadow the underlying problem.

kiview · 2017-12-08T09:00:47Z

core/src/main/java/org/testcontainers/containers/GenericContainer.java

+    protected Set<Integer> getLivenessCheckPorts() {
+        final Set<Integer> result = new HashSet<>();
+        if (exposedPorts.size() > 0) {
+            result.addAll(exposedPorts.stream()


Can we put these blocks into private methods? Something like getExposedPortsAsInt and getBoundPortsAsInt.

kiview · 2017-12-08T09:05:36Z

core/src/main/java/org/testcontainers/containers/GenericContainer.java

-            Set<Link> links = dockerClient.listContainersCmd().exec().stream()
-                    .filter(container -> container.getNames()[0].endsWith(linkableContainer.getContainerName()))
-                    .map(container -> new Link(container.getNames()[0], alias))
+            Set<Link> links = dockerClient.listContainersCmd()


I would like this in a private method as well (since I really have to look into the stream to get whats happening here). Maybe findLinksForX (whatever X is)?

kiview · 2017-12-08T09:09:41Z

core/src/main/java/org/testcontainers/containers/wait/HostPortWaitStrategy.java


-        if (shouldCheckWithCommand()) {
-            List<Integer> exposedPorts = container.getExposedPorts();
+        final Set<Integer> internalPorts = exposedPorts.stream()


Again I would prefer to see a private method, getInternalPorts(). Sorry for being so picky about private methods, that's a coding style I prefer, you don't have to follow if you aren't a fan of it, just think it makes it easier to get a high level understanding of what's happening, especially for people more unfamiliar with the codebase 😁

No, that's a good point so thanks for bringing it up (all three!)

Will fix :)

If only we had Extensions...!

kiview · 2017-12-08T09:16:14Z

@rnorth I just realized in order to fix #455 we'd need something like a WaitForAnyWaitStrategy composite.

rnorth · 2017-12-09T08:40:13Z

@kiview

I just realized in order to fix #455 we'd need something like a WaitForAnyWaitStrategy composite.

yes, we would. Would it be OK with you if we tackle #455 separately, though?

kiview · 2017-12-09T10:44:13Z

@rnorth Of course, clearly a separate feature and concern (the formatting in your last post is off btw. ;) ).

Fix NPE

rnorth requested review from bsideup and kiview November 26, 2017 20:32

bsideup reviewed Nov 26, 2017

View reviewed changes

rnorth force-pushed the wait-robustness branch 2 times, most recently from 496096b to 9c716a8 Compare December 7, 2017 22:30

rnorth changed the title ~~WIP: Wait robustness~~ Wait robustness Dec 7, 2017

rnorth mentioned this pull request Dec 7, 2017

VNC recorder for selenium container creates empty video files #466

Closed

kiview requested changes Dec 8, 2017

View reviewed changes

rnorth added 11 commits December 9, 2017 18:52

WIP - refactor port awaits for improved robustness

e15e30b

WIP - refactor port awaits for improved robustness

df51f3f

Fix NPE

Add tests and fix internal detection command

9f5ecd1

Ripple through refactoring

3a8c7b8

Update CHANGELOG

77fce3a

Codacy cleanup

4fb4ca4

Code review improvements

a336a11

Fix incorrect factory for Sets

e0be5ab

Experimental change

9345d80

Update changelog to reflect probable fix for #466

9be995f

Refactoring port mapping code following review comments

0a313a2

rnorth force-pushed the wait-robustness branch from 4b95228 to 0a313a2 Compare December 9, 2017 19:11

Extract methods for readability

8d64783

kiview approved these changes Dec 9, 2017

View reviewed changes

rnorth merged commit f1ce2b1 into master Dec 10, 2017

rnorth deleted the wait-robustness branch December 10, 2017 08:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wait robustness #504

Wait robustness #504

rnorth commented Nov 26, 2017 •

edited

Loading

bsideup Nov 26, 2017

rnorth Nov 26, 2017

bsideup Nov 26, 2017

rnorth Nov 26, 2017

bsideup Nov 26, 2017

rnorth Nov 26, 2017

bsideup Nov 26, 2017

rnorth Nov 26, 2017

bsideup Nov 26, 2017

bsideup Nov 26, 2017

rnorth Nov 26, 2017

bsideup Nov 26, 2017

rnorth Nov 26, 2017

rnorth commented Nov 26, 2017

rnorth commented Nov 26, 2017

rnorth commented Nov 27, 2017

kiview commented Nov 29, 2017

rnorth commented Nov 29, 2017 via email

kiview commented Nov 30, 2017

kiview Dec 8, 2017

kiview Dec 8, 2017

kiview Dec 8, 2017

rnorth Dec 9, 2017

rnorth Dec 9, 2017

rnorth Dec 9, 2017

kiview commented Dec 8, 2017

rnorth commented Dec 9, 2017 •

edited

Loading

kiview commented Dec 9, 2017

Wait robustness #504

Wait robustness #504

Conversation

rnorth commented Nov 26, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rnorth commented Nov 26, 2017

rnorth commented Nov 26, 2017

rnorth commented Nov 27, 2017

kiview commented Nov 29, 2017

rnorth commented Nov 29, 2017 via email

kiview commented Nov 30, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kiview commented Dec 8, 2017

rnorth commented Dec 9, 2017 • edited Loading

kiview commented Dec 9, 2017

rnorth commented Nov 26, 2017 •

edited

Loading

rnorth commented Dec 9, 2017 •

edited

Loading