Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug: kafka testcontainer does not support adding a network #637

Closed
champialex opened this issue Jul 4, 2024 · 2 comments · Fixed by #638 or #639
Closed

Bug: kafka testcontainer does not support adding a network #637

champialex opened this issue Jul 4, 2024 · 2 comments · Fixed by #638 or #639

Comments

@champialex
Copy link

champialex commented Jul 4, 2024

Describe the bug

KafkaContainer.tc_start only allows containers to have a single IP, so it is not possible to add a network

Changing
listeners = f"PLAINTEXT://{host}:{port},BROKER://$(hostname -i):9092" to
listeners = f"PLAINTEXT://{host}:{port},BROKER://$(hostname -i | cut -d' ' -f1):9092" fixes it

Exception in the container logs:

java.lang.IllegalArgumentException: Error creating broker listeners from 'PLAINTEXT://192.168.64.2:33647,BROKER://172.17.0.3 172.21.0.2:9092': Unable to parse BROKER://172.17.0.3 172.21.0.2:9092 to a broker endpoint
	at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:266)
	at kafka.server.KafkaConfig.effectiveAdvertisedListeners(KafkaConfig.scala:2154)
	at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:2275)
	at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:2233)
	at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1604)
	at kafka.Kafka$.buildServer(Kafka.scala:72)
	at kafka.Kafka$.main(Kafka.scala:91)
	at kafka.Kafka.main(Kafka.scala)
Caused by: org.apache.kafka.common.KafkaException: Unable to parse BROKER://172.17.0.3 172.21.0.2:9092 to a broker endpoint
	at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:58)
	at kafka.utils.CoreUtils$.$anonfun$listenerListToEndPoints$10(CoreUtils.scala:263)
	at scala.collection.StrictOptimizedIterableOps.map(StrictOptimizedIterableOps.scala:100)
	at scala.collection.StrictOptimizedIterableOps.map$(StrictOptimizedIterableOps.scala:87)
	at scala.collection.mutable.ArraySeq.map(ArraySeq.scala:37)
	at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:263)
	... 7 more

To Reproduce

    from kafka import KafkaAdminClient
    from testcontainers.kafka import KafkaContainer

    with Network() as network:
        kafka_ctr = KafkaContainer()
        kafka_ctr.with_network(network)
        kafka_ctr.with_network_aliases("kafka")

        with kafka_ctr:
            print("started") # Will not reach here and timeout
            admin_client = KafkaAdminClient(bootstrap_servers=[kafka_ctr.get_bootstrap_server()])
            print(admin_client.describe_cluster())

Runtime environment

[[package]]
name = "testcontainers"
version = "4.7.1"

# Get the operating system information (on a unix os).
$ uname -a
Darwin sydmacg366 23.3.0 Darwin Kernel Version 23.3.0: Wed Dec 20 21:31:10 PST 2023; root:xnu-10002.81.5~7/RELEASE_ARM64_T6031 arm64
# Get the python version.
$ python --version
Python 3.12.2
# Get the docker version and other docker information.
$ docker info

Client: Docker Engine - Community
 Version:    27.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.15.1
    Path:     /Users/adecham/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  2.28.1
    Path:     /Users/adecham/.docker/cli-plugins/docker-compose

Server:
 Containers: 338
  Running: 1
  Paused: 0
  Stopped: 337
 Images: 24
 Server Version: 23.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 0cae528dd6cb557f7201036e9f43420650207b58
 runc version: 860f061b76bb4fc671f0f9e900f7d80ff93d4eb7
 init version:
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 6.1.55-0-virt
 Operating System: Alpine Linux v3.18
 OSType: linux
 Architecture: aarch64
 CPUs: 2
 Total Memory: 7.753GiB
 Name: colima
 ID: c74f716c-594e-4e7c-8ae1-f9f71efbd2ac
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
@alexanderankin
Copy link
Member

seems related to #633

can you share more of the use case, i understand this is a complete reproduction but i am also kind of curious the use case, whether in python form or a general description - I do not use this feature myself so I want to understand a bit better. thanks.

@champialex
Copy link
Author

Thanks for the quick turnaround!

The use case is when chaining several containers (in my case telegraf and kafka). Having a network alias makes it possible to do that without having to do complicated file generation based on actual IPs: I can just point telegraf to kafka:9092.

alexanderankin pushed a commit that referenced this issue Jul 15, 2024
🤖 I have created a release *beep* *boop*
---


##
[4.7.2](testcontainers-v4.7.1...testcontainers-v4.7.2)
(2024-07-15)


### Bug Fixes

* Add container Trino
([#642](#642))
([49ce5a5](49ce5a5)),
closes
[#641](#641)
* **core:** Improve private registry support (tolerate not implemented
fields in DOCKER_AUTH_CONFIG)
([#647](#647))
([766c382](766c382))
* **kafka:** add a flag to limit to first hostname for use with networks
([#638](#638))
([0ce4fec](0ce4fec)),
closes
[#637](#637)
* **modules:** Mailpit container base API URL helper method
([#643](#643))
([df07586](df07586))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants