Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Docker networks. #372

Merged
merged 13 commits into from
Jun 25, 2017
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ All notable changes to this project will be documented in this file.
### Fixed

### Changed
- Added support for Docker networks (#372)

## [1.3.1] - 2017-06-22
### Fixed
Expand Down
18 changes: 18 additions & 0 deletions core/src/main/java/org/testcontainers/containers/Container.java
Original file line number Diff line number Diff line change
Expand Up @@ -209,6 +209,24 @@ default void addFileSystemBind(final String hostPath, final String containerPath
*/
SELF withNetworkMode(String networkMode);

/**
* Set the network for this container, similar to the <code>--network &lt;name&gt;</code>
* option on the docker CLI.
*
* @param network the instance of {@link Network}
* @return this
*/
SELF withNetwork(Network network);

/**
* Set the network aliases for this container, similar to the <code>--network-alias &lt;my-service&gt;</code>
* option on the docker CLI.
*
* @param aliases the list of aliases
* @return this
*/
SELF withNetworkAliases(String... aliases);

/**
* Map a resource (file or directory) on the classpath to a path inside the container.
* This will only work if you are running your tests outside a Docker container.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,12 @@ public class GenericContainer<SELF extends GenericContainer<SELF>>
@NonNull
private String networkMode;

@NonNull
private Network network;

@NonNull
private List<String> networkAliases = new ArrayList<>();

@NonNull
private Future<String> image;

Expand Down Expand Up @@ -412,7 +418,10 @@ private void applyConfiguration(CreateContainerCmd createCommand) {
.toArray(String[]::new);
createCommand.withExtraHosts(extraHostsArray);

if (networkMode != null) {
if (network != null) {
createCommand.withNetworkMode(network.getId());
createCommand.withAliases(this.networkAliases);
} else if (networkMode != null) {
createCommand.withNetworkMode(networkMode);
}

Expand Down Expand Up @@ -615,6 +624,18 @@ public SELF withNetworkMode(String networkMode) {
return self();
}

@Override
public SELF withNetwork(Network network) {
this.network = network;
return self();
}

@Override
public SELF withNetworkAliases(String... aliases) {
Collections.addAll(this.networkAliases, aliases);
return self();
}

/**
* {@inheritDoc}
*/
Expand Down
107 changes: 107 additions & 0 deletions core/src/main/java/org/testcontainers/containers/Network.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
package org.testcontainers.containers;

import com.github.dockerjava.api.command.CreateNetworkCmd;
import lombok.*;
import org.junit.rules.ExternalResource;
import org.junit.rules.TestRule;
import org.testcontainers.DockerClientFactory;
import org.testcontainers.utility.ResourceReaper;

import java.util.LinkedHashSet;
import java.util.Set;
import java.util.UUID;
import java.util.concurrent.atomic.AtomicReference;
import java.util.function.Consumer;
import java.util.function.Supplier;

public interface Network extends AutoCloseable, TestRule {

String getId();

String getName();

Boolean getEnableIpv6();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you really want a Boolean?


String getDriver();

boolean isCreated();

default boolean create() {
return getId() != null;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would leave out the default implementation- it's too magic: you need to guess that getId is doing the actual create.
Regarding the Boolean - why not fail if can't create? In most chances you can't continue without a network.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

default implementation here assumes that there is kinda no need to create a network. If id is known then what else to "create"?

The method returns boolean instead of an exception because it's up to the consumer how to react on that. However, I agree that before should throw if it returns false

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Personally, I prefer the previous approach with .as() where the contract is clear, but for now we decided to go with an old approach until 2.0

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the impl you placed the logic of creating the network in getId() - that was my point. In the default implementation you assumed getId will be called this create the network if needed. Am I missing something?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think @asafm point is, that getId() method here is definitly not a regular getter as one might expect, since it does the actual creation as well. So maybe it's having elegant code leveraging Lombok vs. self-explanatory code at this point.

Copy link
Member Author

@bsideup bsideup Jun 24, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

id is the only thing needed to use this network. Default implementation assumes "no creation needed" semantic and returns true if id exists. Do not read the implementation to understand the idea of default implementation here.

Example:

class ExistingNetwork implements Network {
    String getId() {
        return "some-predefined-id";
    }
}

}

@Override
default void close() {
if (isCreated()) {
ResourceReaper.instance().removeNetworks(getName());
}
}

@SneakyThrows
@SuppressWarnings("unchecked")
default <T extends Network> T as(Class<T> type) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious to that purpose of this class, which is related to another question I had: why have an interface? Do we plan on users extending and supplying their own Network implementation?

return type.getDeclaredConstructor(Network.class).newInstance(this);
}

static Network newNetwork() {
return builder().build();
}

static NetworkImpl.NetworkImplBuilder builder() {
return NetworkImpl.builder();
}

@Builder
@Getter
class NetworkImpl extends ExternalResource implements Network {

private final String name = UUID.randomUUID().toString();

private Boolean enableIpv6;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same thing with the Boolean here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

null here means "the value wasn't provided and withEnableIpv6 should not be called (use a default value from CreateNetworkCmd)


private String driver;

@Singular
private Set<Consumer<CreateNetworkCmd>> createNetworkCmdModifiers = new LinkedHashSet<>();

@Getter(lazy = true)
private final String id = ((Supplier<String>) () -> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why a variable of type String receives a lambda function, casted to a Suplier of String and then executed? I mean - what's going on?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lazy = true is the answer

ResourceReaper.instance().registerNetworkForCleanup(getName());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer to have this in a private method.


CreateNetworkCmd createNetworkCmd = DockerClientFactory.instance().client().createNetworkCmd();

createNetworkCmd.withName(getName());
createNetworkCmd.withCheckDuplicate(true);

if (getEnableIpv6() != null) {
createNetworkCmd.withEnableIpv6(getEnableIpv6());
}

if (getDriver() != null) {
createNetworkCmd.withDriver(getDriver());
}

for (Consumer<CreateNetworkCmd> consumer : createNetworkCmdModifiers) {
consumer.accept(createNetworkCmd);
}

return createNetworkCmd.exec().getId();
}).get();

@Override
public boolean isCreated() {
// Lombok with @Getter(lazy = true) will use AtomicReference as a field type for id
return ((AtomicReference<String>) (Object) id).get() != null;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a personal opinion - that's too much magic for such a simple thing as having a field called Id of type String. Maybe it will more clear if for this field, Lombok won't be used.

}

@Override
protected void before() throws Throwable {
create();
}

@Override
protected void after() {
close();
}
}
}
43 changes: 26 additions & 17 deletions core/src/main/java/org/testcontainers/utility/ResourceReaper.java
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,12 @@
import com.github.dockerjava.api.exception.DockerException;
import com.github.dockerjava.api.exception.InternalServerErrorException;
import com.github.dockerjava.api.exception.NotFoundException;
import com.github.dockerjava.api.model.Container;
import com.github.dockerjava.api.model.Network;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.testcontainers.DockerClientFactory;

import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.*;
import java.util.concurrent.ConcurrentHashMap;

/**
Expand All @@ -25,7 +22,7 @@ public final class ResourceReaper {
private static ResourceReaper instance;
private final DockerClient dockerClient;
private Map<String, String> registeredContainers = new ConcurrentHashMap<>();
private List<String> registeredNetworks = new ArrayList<>();
private Set<String> registeredNetworks = Collections.newSetFromMap(new ConcurrentHashMap<>());

private ResourceReaper() {
dockerClient = DockerClientFactory.instance().client();
Expand Down Expand Up @@ -145,22 +142,34 @@ public void removeNetworks(String identifier) {
}

private void removeNetwork(String networkName) {
List<Network> networks;
try {
networks = dockerClient.listNetworksCmd().withNameFilter(networkName).exec();
} catch (DockerException e) {
LOGGER.trace("Error encountered when looking up network for removal (name: {}) - it may not have been removed", networkName);
return;
}
try {
// First try to remove by name
dockerClient.removeNetworkCmd(networkName).exec();
} catch (Exception e) {
LOGGER.trace("Error encountered removing network by name ({}) - it may not have been removed", networkName);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two tests running with same network name will fail , at least the second create will. They will not understand why since the failure is hidden under trace level. I would keep this at error level or warn.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

network's name is random (see NetworkImpl)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So no way of allowing me to specify a name? I'm Starting to build Testclusters, where I'm reusing containers across runs to shorten test duration. My assumption is same network name across runs. Would love to use the class created here for this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If your use case would reuse the containers, it would reuse the network as well, I assume?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@asafm we don't allow users to set names of containers (and now networks) because it might lead to the names conflict (especially with parallel execution).

As @kiview said: if you reuse things, reuse the networks also

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm reusing across builds, on same machine. Quite similar to docker compose. I can code it with the client my self, but I think it's harmless to leave the opening to set the name to who ever needs it. I'm ok with any direction you take.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's think about whether we could allow reuse between runs separately. It's probably quite doable, but for the sake of this PR let's resume later.

}

for (Network network : networks) {
List<Network> networks;
try {
dockerClient.removeNetworkCmd(network.getId()).exec();
registeredNetworks.remove(network.getId());
LOGGER.debug("Removed network: {}", networkName);
} catch (DockerException e) {
LOGGER.trace("Error encountered removing network (name: {}) - it may not have been removed", network.getName());
// Then try to list all networks with the same name
networks = dockerClient.listNetworksCmd().withNameFilter(networkName).exec();
} catch (Exception e) {
LOGGER.trace("Error encountered when looking up network for removal (name: {}) - it may not have been removed", networkName);
return;
}

for (Network network : networks) {
try {
dockerClient.removeNetworkCmd(network.getId()).exec();
registeredNetworks.remove(network.getId());
LOGGER.debug("Removed network: {}", networkName);
} catch (Exception e) {
LOGGER.trace("Error encountered removing network (name: {}) - it may not have been removed", network.getName());
}
}
} finally {
registeredNetworks.remove(networkName);
}
}
}
105 changes: 105 additions & 0 deletions core/src/test/java/org/testcontainers/containers/NetworkTest.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
package org.testcontainers.containers;

import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.runners.Enclosed;
import org.junit.runner.RunWith;
import org.testcontainers.DockerClientFactory;

import static org.rnorth.visibleassertions.VisibleAssertions.*;
import static org.testcontainers.containers.Network.newNetwork;

@RunWith(Enclosed.class)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Neat - I didn't know about this!

public class NetworkTest {

public static class WithRules {

@Rule
public Network network = newNetwork();

@Rule
public GenericContainer foo = new GenericContainer()
.withNetwork(network)
.withNetworkAliases("foo")
.withCommand("/bin/sh", "-c", "while true ; do printf 'HTTP/1.1 200 OK\\n\\nyay' | nc -l -p 8080; done");

@Rule
public GenericContainer bar = new GenericContainer()
.withNetwork(network)
.withCommand("top");

@Test
public void testNetworkSupport() throws Exception {
String response = bar.execInContainer("wget", "-O", "-", "http://foo:8080").getStdout();
assertEquals("received response", "yay", response);
}
}

public static class WithoutRules {

@Test
public void testNetworkSupport() throws Exception {
try (
Network network = newNetwork();

GenericContainer foo = new GenericContainer()
.withNetwork(network)
.withNetworkAliases("foo")
.withCommand("/bin/sh", "-c", "while true ; do printf 'HTTP/1.1 200 OK\\n\\nyay' | nc -l -p 8080; done");

GenericContainer bar = new GenericContainer()
.withNetwork(network)
.withCommand("top")
) {
foo.start();
bar.start();

String response = bar.execInContainer("wget", "-O", "-", "http://foo:8080").getStdout();
assertEquals("received response", "yay", response);
}
}

@Test
public void testBuilder() throws Exception {
try (
Network network = Network.builder()
.driver("macvlan")
.build();
) {
network.create();
assertEquals(
"Flag is set",
"macvlan",
DockerClientFactory.instance().client().inspectNetworkCmd().withNetworkId(network.getName()).exec().getDriver()
);
}
}

@Test
public void testModifiers() throws Exception {
try (
Network network = Network.builder()
.createNetworkCmdModifier(cmd -> cmd.withDriver("macvlan"))
.build();
) {
network.create();
assertEquals(
"Flag is set",
"macvlan",
DockerClientFactory.instance().client().inspectNetworkCmd().withNetworkId(network.getName()).exec().getDriver()
);
}
}

@Test
public void testLaziness() throws Exception {
try (
Network network = newNetwork()
) {
assertFalse("Not created by default", network.isCreated());
assertNotNull("Returns an id", network.getId());
assertTrue("Is created after id request", network.isCreated());
}
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.testcontainers.containers.BrowserWebDriverContainer;
import org.testcontainers.containers.Network;
import org.testcontainers.containers.NginxContainer;

import java.io.*;
Expand All @@ -23,13 +24,18 @@ public class LinkedContainerTest {
private static File contentFolder = new File(System.getProperty("user.home") + "/.tmp-test-container");

@Rule
public NginxContainer nginx = new NginxContainer()
public Network network = Network.newNetwork();

@Rule
public NginxContainer nginx = new NginxContainer<>()
.withNetwork(network)
.withNetworkAliases("nginx")
.withCustomContent(contentFolder.toString());

@Rule
public BrowserWebDriverContainer chrome = new BrowserWebDriverContainer()
.withDesiredCapabilities(DesiredCapabilities.chrome())
.withLinkToContainer(nginx, "nginx");
public BrowserWebDriverContainer chrome = new BrowserWebDriverContainer<>()
.withNetwork(network)
.withDesiredCapabilities(DesiredCapabilities.chrome());

@BeforeClass
public static void setupContent() throws FileNotFoundException {
Expand Down