[Bug]: Couchbase containers intermittently hang on startup

[couchbaseEd]

Testcontainers version

0.32.0

Using the latest Testcontainers version?

Yes

Host OS

macOS Sonoma

Host arch

ARM

Go version

1.22.2

Docker version

Client:
 Version:           27.0.3
 API version:       1.46
 Go version:        go1.21.11
 Git commit:        7d4bcd8
 Built:             Fri Jun 28 23:59:41 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.32.0 (157355)
 Engine:
  Version:          27.0.3
  API version:      1.46 (minimum version 1.24)
  Go version:       go1.21.11
  Git commit:       662f78c
  Built:            Sat Jun 29 00:02:44 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.7.18
  GitCommit:        ae71819c4f5e67bb4d5ae76a6b735f29cc25774e
 runc:
  Version:          1.7.18
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

Client:
 Version:    27.0.3
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.15.1-desktop.1
    Path:     /Users/edwardmiller/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.28.1-desktop.1
    Path:     /Users/edwardmiller/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.32
    Path:     /Users/edwardmiller/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Alpha) (Docker Inc.)
    Version:  v0.0.14
    Path:     /Users/edwardmiller/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/edwardmiller/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.25
    Path:     /Users/edwardmiller/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/edwardmiller/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.3.0
    Path:     /Users/edwardmiller/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/edwardmiller/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.10.0
    Path:     /Users/edwardmiller/.docker/cli-plugins/docker-scout

Server:
 Containers: 9
  Running: 0
  Paused: 0
  Stopped: 9
 Images: 199
 Server Version: 27.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae71819c4f5e67bb4d5ae76a6b735f29cc25774e
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.6.32-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 10
 Total Memory: 11.67GiB
 Name: docker-desktop
 ID: 105e61c8-acfb-4b76-b7c3-35c6e637909b
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/edwardmiller/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

What happened?

Sometimes the creation of a Couchbase container hangs. When making the call to create the primary index the response isn't checked. This means that any errors returned in the response body are not caught and hence the isPrimaryIndexOnline polls until the timeout. In our case this occurs if we try to create the index to early before the bucket is ready. See output from a test app with some print debugging below.

Relevant log output

starting setup
Creating primary index
{
"requestID": "fbb6b590-e549-4c3b-953b-4b35d3f8abd0",
"signature": null,
"results": [
],
"errors": [{"code":5000,"msg":"GSI CreatePrimaryIndex() - cause: Index creation for index #primary, bucket nutshell, scope _default, collection _default cannot start. Reason: Create index or Alter replica cannot proceed due to rebalance in progress..","reason":{"_level":"exception","caller":"secondary_index:413","cause":{"error":"Index creation for index #primary, bucket nutshell, scope _default, collection _default cannot start.","reason":"Create index or Alter replica cannot proceed due to rebalance in progress..","source":"CreatePrimaryIndex()"},"code":4350,"icause":"Index creation for index #primary, bucket nutshell, scope _default, collection _default cannot start. Reason: Create index or Alter replica cannot proceed due to rebalance in progress..","key":"indexing.error","message":"GSI CreatePrimaryIndex()"}}],
"status": "fatal",
"metrics": {"elapsedTime": "32.491ms","executionTime": "32.454959ms","resultCount": 0,"resultSize": 0,"serviceLoad": 2,"errorCount": 1}
}

is index online?: false
is index online?: false
is index online?: false
is index online?: false
is index online?: false
is index online?: false
is index online?: false
is index online?: false
^Csignal: interrupt

package main

import (
	"context"
	"fmt"
	"os"
	"time"

	"github.com/couchbase/gocb/v2"
	tcouchbase "github.com/testcontainers/testcontainers-go/modules/couchbase"
)

// Config used in couchbase connection
type CouchbaseConfig struct {
	Endpoint string
	Username string
	Password string
}

type cleanupFunc func()

func main() {

	ctx := context.Background()
	_, cleanup, err := Setup(ctx, "bucketname")
	if err != nil {
		fmt.Println(err)
		os.Exit(1)
	}
	defer cleanup()
	fmt.Println("done")
}

// Setup creates a new instance of a couchbase cluster for testing
func Setup(ctx context.Context, bucketName string) (*gocb.Cluster, cleanupFunc, error) {
	fmt.Println("starting setup")
	bucket := tcouchbase.NewBucket(bucketName).
		WithQuota(100).
		WithReplicas(0).
		WithFlushEnabled(true).
		WithPrimaryIndex(true)
	couchbaseContainer, err := tcouchbase.Run(ctx,
		"couchbase:enterprise-7.6.1",
		tcouchbase.WithBuckets(bucket),
	)
	if err != nil {
		return nil, nil, err
	}

	connStr, err := couchbaseContainer.ConnectionString(ctx)
	if err != nil {
		return nil, nil, err
	}

	cluster, err := NewCluster(CouchbaseConfig{
		Endpoint: connStr,
		Username: couchbaseContainer.Username(),
		Password: couchbaseContainer.Password(),
	})
	if err != nil {
		return nil, nil, err
	}

	cleanup := func() {
		_ = couchbaseContainer.Terminate(ctx)
	}

	return cluster, cleanup, nil
}

// Takes in a config and creates a couchbase cluster
func NewCluster(cfg CouchbaseConfig) (*gocb.Cluster, error) {
	timeoutConnection := 10 * time.Second
	timeoutKV := 250 * time.Millisecond
	timeoutQuery := 2 * time.Second
	options := gocb.ClusterOptions{
		Authenticator: gocb.PasswordAuthenticator{
			Username: cfg.Username,
			Password: cfg.Password,
		},
		TimeoutsConfig: gocb.TimeoutsConfig{
			ConnectTimeout: timeoutConnection,
			KVTimeout:      timeoutKV,
			QueryTimeout:   timeoutQuery,
		},
	}

	couchbaseCluster, err := gocb.Connect(cfg.Endpoint, options)
	if err != nil {
		return nil, fmt.Errorf("unable to set up couchbase client: %w", err)
	}

	return couchbaseCluster, nil
}

Additional information

No response

[mdelapenya]

Hi @couchbaseEd thanks for raising this issue. Do you feel the wait strategy in the container should be refined to cover that use case?

Are you working for couchbase btw? That would help a lot in creating the perfect experience for the module

[couchbaseEd]

Hi yes I do work for Couchbase. Ive got a pr which just checks the response for an error and adds a backoff similar to the isPrimaryIndexOnline which seems to do the trick. Just about to put it up for review now.

[mdelapenya]

Could this also be related? testcontainers/testcontainers-java#7049

[couchbaseEd]

It could well be but its difficult to know without actually seeing the response. The issue is the cluster does an initial rebalance on start up when adding services which needs to finish before we can do anything with it.