[Bug]: Couchbase container hangs during startup on subsequent launches with reuse enabled

[monofunc]

Testcontainers version

4.1.0

Using the latest Testcontainers version?

Yes

Host OS

macOS

Host arch

ARM

.NET version

9.0.101

Docker version

Client:
 Version:           27.4.0
 API version:       1.47
 Go version:        go1.22.10
 Git commit:        bde2b89
 Built:             Sat Dec  7 10:35:43 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.37.2 (179585)
 Engine:
  Version:          27.4.0
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.10
  Git commit:       92a8393
  Built:            Sat Dec  7 10:38:33 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.7.21
  GitCommit:        472731909fa34bd7bc9c087e4c27943f9835f111
 runc:
  Version:          1.1.13
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

Client:
 Version:    27.4.0
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  ai: Ask Gordon - Docker Agent (Docker Inc.)
    Version:  v0.5.1
    Path:     /Users/mono/.docker/cli-plugins/docker-ai
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.19.2-desktop.1
    Path:     /Users/mono/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.31.0-desktop.2
    Path:     /Users/mono/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.37
    Path:     /Users/mono/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Beta) (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/mono/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/mono/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.27
    Path:     /Users/mono/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/mono/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     /Users/mono/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/mono/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.15.1
    Path:     /Users/mono/.docker/cli-plugins/docker-scout

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 27.4.0
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.10.14-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 10
 Total Memory: 7.654GiB
 Name: docker-desktop
 ID: c6c6b58d-cbb2-403b-b91a-51269cbf4646
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/mono/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

What happened?

When attempting to reuse a container, it hangs indefinitely at the message "Couchbase container is starting, performing configuration" during startup.

The issue occurs because credentials configured during the first container start persist in the reused container, but WaitUntilNodeIsReady and all subsequent configuration requests are sent without authentication, which fail with Unauthorized errors.

Relevant log output

Additional information

public class ReusableCouchbaseContainerTest
{
    [Fact]
    public async Task ReuseCouchbaseContainer()
    {
        var newContainer = new CouchbaseBuilder()
            .WithImage("couchbase:community-7.6.2")
            .WithReuse(true)
            .Build();

        await newContainer.StartAsync();

        var newId = newContainer.Id;

        await newContainer.StopAsync();

        await Task.Delay(TimeSpan.FromSeconds(1));

        await newContainer.DisposeAsync();

        var existingContainer = new CouchbaseBuilder()
            .WithImage("couchbase:community-7.6.2")
            .WithReuse(true)
            .Build();

        await existingContainer.StartAsync();

        var existingId = existingContainer.Id;

        Assert.Equal(newId, existingId);
    }
}

[HofmeisterAn]

I haven't looked closely into the sources yet and haven't tried the reproducer.

The issue occurs because credentials configured during the first container start persist in the reused container, but WaitUntilNodeIsReady and all subsequent configuration requests are sent without authentication, which fail with Unauthorized errors.

What makes you think that this is the issue Ok, I think we are talking about the same. You did not mean the wait strategies 👍? The wait strategies contain the credentials. I think the issue is that we run the startup callback ConfigureCouchbaseAsync. Can you try to "unset" the startup callback for your second configuration?

var existingContainer = new CouchbaseBuilder()
    .WithImage("couchbase:community-7.6.2")
    .WithReuse(true)
    .WithStartupCallback((_, _) => Task.CompletedTask)
    .Build();

[monofunc]

Yes, skipping the configuration works, but we can't detect if a container needs initial setup or trigger when needed, e.g. when using the container as a collection fixture in xUnit on subsequent launches.

In my case, I'd like to use a single database for running all tests located in different assemblies, since Couchbase startup time is quite long.

[HofmeisterAn]

Yes, skipping the configuration works, but we can't detect if a container needs initial setup [...]

Thanks. That's what I thought, I'm not sure how we could support this with the Couchbase module. It's a tricky issue to detect. If we come up with a solution that works, I'm happy to adjust the module accordingly.

In my case, I'd like to use a single database for running all tests located in different assemblies, since Couchbase startup time is quite long.

I understand your goal, and it makes sense to reduce the overhead caused by the long Couchbase startup time. However, I'd like to point out that this approach may not be ideal. It could lead to resource leaks, as reusing containers disables the Resource Reaper. Perhaps using a singleton for each test project could be a trade-off (or use a single test project for these kind of tests).