failed to create new hosts file: unable to replace "host-gateway" of host entry "host.k3d.internal:host-gateway": host containers internal IP address is empty

[eugene-chernyshenko]

What did you do

k3d cluster create test --verbose --trace --no-rollback

Terminal output

DEBU[0000] DOCKER_SOCK=/var/run/docker.sock             
DEBU[0000] Runtime Info:
&{Name:docker Endpoint:/var/run/docker.sock Version:4.7.0 OSType:linux OS:fedora Arch:arm64 CgroupVersion:2 CgroupDriver:systemd Filesystem:xfs InfoName:localhost.localdomain} 
DEBU[0000] Additional CLI Configuration:
cli:
  api-port: ""
  env: []
  k3s-node-labels: []
  k3sargs: []
  ports: []
  registries:
    create: ""
  runtime-labels: []
  runtime-ulimits: []
  volumes: []
hostaliases: [] 
DEBU[0000] Configuration:
agents: 0
image: docker.io/rancher/k3s:v1.27.4-k3s1
network: ""
options:
  k3d:
    disableimagevolume: false
    disableloadbalancer: false
    disablerollback: true
    loadbalancer:
      configoverrides: []
    timeout: 0s
    wait: true
  kubeconfig:
    switchcurrentcontext: true
    updatedefaultkubeconfig: true
  runtime:
    agentsmemory: ""
    gpurequest: ""
    hostpidmode: false
    serversmemory: ""
registries:
  config: ""
  use: []
servers: 1
subnet: ""
token: "" 
TRAC[0000] Trying to read config apiVersion='k3d.io/v1alpha5', kind='simple' 
DEBU[0000] ========== Simple Config ==========
{TypeMeta:{Kind:Simple APIVersion:k3d.io/v1alpha5} ObjectMeta:{Name:} Servers:1 Agents:0 ExposeAPI:{Host: HostIP: HostPort:} Image:docker.io/rancher/k3s:v1.27.4-k3s1 Network: Subnet: ClusterToken: Volumes:[] Ports:[] Options:{K3dOptions:{Wait:true Timeout:0s DisableLoadbalancer:false DisableImageVolume:false NoRollback:true NodeHookActions:[] Loadbalancer:{ConfigOverrides:[]}} K3sOptions:{ExtraArgs:[] NodeLabels:[]} KubeconfigOptions:{UpdateDefaultKubeconfig:true SwitchCurrentContext:true} Runtime:{GPURequest: ServersMemory: AgentsMemory: HostPidMode:false Labels:[] Ulimits:[]}} Env:[] Registries:{Use:[] Create:<nil> Config:} HostAliases:[]}
========================== 
TRAC[0000] VolumeFilterMap: map[]                       
TRAC[0000] PortFilterMap: map[]                         
TRAC[0000] K3sNodeLabelFilterMap: map[]                 
TRAC[0000] RuntimeLabelFilterMap: map[]                 
TRAC[0000] EnvFilterMap: map[]                          
DEBU[0000] ========== Merged Simple Config ==========
{TypeMeta:{Kind:Simple APIVersion:k3d.io/v1alpha5} ObjectMeta:{Name:} Servers:1 Agents:0 ExposeAPI:{Host: HostIP: HostPort:57915} Image:docker.io/rancher/k3s:v1.27.4-k3s1 Network: Subnet: ClusterToken: Volumes:[] Ports:[] Options:{K3dOptions:{Wait:true Timeout:0s DisableLoadbalancer:false DisableImageVolume:false NoRollback:true NodeHookActions:[] Loadbalancer:{ConfigOverrides:[]}} K3sOptions:{ExtraArgs:[] NodeLabels:[]} KubeconfigOptions:{UpdateDefaultKubeconfig:true SwitchCurrentContext:true} Runtime:{GPURequest: ServersMemory: AgentsMemory: HostPidMode:false Labels:[] Ulimits:[]}} Env:[] Registries:{Use:[] Create:<nil> Config:} HostAliases:[]}
========================== 
DEBU[0000] generated loadbalancer config:
ports:
  6443.tcp:
  - k3d-test-server-0
settings:
  workerConnections: 1024 
DEBU[0000] ===== Merged Cluster Config =====
&{TypeMeta:{Kind: APIVersion:} Cluster:{Name:test Network:{Name:k3d-test ID: External:false IPAM:{IPPrefix:invalid Prefix IPsUsed:[] Managed:false} Members:[]} Token: Nodes:[0x140005a3a00 0x140005a3ba0] InitNode:<nil> ExternalDatastore:<nil> KubeAPI:0x140003594c0 ServerLoadBalancer:0x1400033b920 ImageVolume: Volumes:[]} ClusterCreateOpts:{DisableImageVolume:false WaitForServer:true Timeout:0s DisableLoadBalancer:false GPURequest: ServersMemory: AgentsMemory: NodeHooks:[] GlobalLabels:map[app:k3d] GlobalEnv:[] HostAliases:[] Registries:{Create:<nil> Use:[] Config:<nil>}} KubeconfigOpts:{UpdateDefaultKubeconfig:true SwitchCurrentContext:true}}
===== ===== ===== 
DEBU[0000] '--kubeconfig-update-default set: enabling wait-for-server 
INFO[0000] Prep: Network                                
INFO[0000] Created network 'k3d-test'                   
INFO[0000] Created image volume k3d-test-images         
TRAC[0000] Using Registries: []                         
TRAC[0000] 
===== Creating Cluster =====

Runtime:
{}

Cluster:
&{Name:test Network:{Name:k3d-test ID:adaa120f35715aecf62ae28c14869e43b553c99c1625eb69ece8877fbb9fb0da External:false IPAM:{IPPrefix:10.89.1.0/24 IPsUsed:[] Managed:false} Members:[]} Token: Nodes:[0x140005a3a00 0x140005a3ba0] InitNode:<nil> ExternalDatastore:<nil> KubeAPI:0x140003594c0 ServerLoadBalancer:0x1400033b920 ImageVolume:k3d-test-images Volumes:[k3d-test-images]}

ClusterCreatOpts:
&{DisableImageVolume:false WaitForServer:true Timeout:0s DisableLoadBalancer:false GPURequest: ServersMemory: AgentsMemory: NodeHooks:[] GlobalLabels:map[app:k3d k3d.cluster.imageVolume:k3d-test-images k3d.cluster.network:k3d-test k3d.cluster.network.external:false k3d.cluster.network.id:adaa120f35715aecf62ae28c14869e43b553c99c1625eb69ece8877fbb9fb0da k3d.cluster.network.iprange:10.89.1.0/24] GlobalEnv:[] HostAliases:[] Registries:{Create:<nil> Use:[] Config:<nil>}}

============================
	 
TRAC[0000] Docker Machine not specified via DOCKER_MACHINE_NAME env var 
TRAC[0000] [Docker] Not using docker-machine            
DEBU[0000] [Docker] DockerHost: '' (unix:///Users/eugene/.local/share/containers/podman/machine/qemu/podman.sock) 
INFO[0000] Starting new tools node...                   
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock             
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock             
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock             
DEBU[0000] Detected CgroupV2, enabling custom entrypoint (disable by setting K3D_FIX_CGROUPV2=false) 
TRAC[0000] Creating node from spec
&{Name:k3d-test-tools Role:noRole Image:ghcr.io/k3d-io/k3d-tools:5.6.0 Volumes:[k3d-test-images:/k3d/images /var/run/docker.sock:/var/run/docker.sock] Env:[] Cmd:[] Args:[noop] Ports:map[] Restart:false Created: HostPidMode:false RuntimeLabels:map[app:k3d k3d.cluster:test k3d.version:v5.6.0] RuntimeUlimits:[] K3sNodeLabels:map[] Networks:[k3d-test] ExtraHosts:[host.k3d.internal:host-gateway] ServerOpts:{IsInit:false KubeAPI:<nil>} AgentOpts:{} GPURequest: Memory: State:{Running:false Status: Started:} IP:{IP:invalid IP Static:false} HookActions:[]} 
TRAC[0000] Creating docker container with translated config
&{ContainerConfig:{Hostname:k3d-test-tools Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[] Tty:false OpenStdin:false StdinOnce:false Env:[K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml] Cmd:[noop] Healthcheck:<nil> ArgsEscaped:false Image:ghcr.io/k3d-io/k3d-tools:5.6.0 Volumes:map[] WorkingDir: Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[app:k3d k3d.cluster:test k3d.role:noRole k3d.version:v5.6.0] StopSignal: StopTimeout:<nil> Shell:[]} HostConfig:{Binds:[k3d-test-images:/k3d/images /var/run/docker.sock:/var/run/docker.sock] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode:bridge PortBindings:map[] RestartPolicy:{Name: MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] ConsoleSize:[0 0] Annotations:map[] CapAdd:[] CapDrop:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[host.k3d.internal:host-gateway] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:true PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[/run: /var/run:] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] MaskedPaths:[] ReadonlyPaths:[] Init:0x140001547ea} NetworkingConfig:{EndpointsConfig:map[k3d-test:0x1400011a240]}} 
DEBU[0000] Created container k3d-test-tools (ID: bf3c10044a664678cd1e994d06dc35c292f3b218b9cdbf7af410038b64aa93d2) 
DEBU[0000] Node k3d-test-tools Start Time: 2023-11-25 01:57:26.779781 +0300 MSK m=+0.206143543 
TRAC[0000] Starting node 'k3d-test-tools'               
INFO[0000] Starting Node 'k3d-test-tools'               
ERRO[0000] Failed to run tools container for cluster 'test' 
INFO[0001] Creating node 'k3d-test-server-0'            
TRAC[0001] Creating node from spec
&{Name:k3d-test-server-0 Role:server Image:docker.io/rancher/k3s:v1.27.4-k3s1 Volumes:[k3d-test-images:/k3d/images] Env:[K3S_TOKEN=VleGSYfkgIMqsIxqKDJU] Cmd:[] Args:[] Ports:map[] Restart:true Created: HostPidMode:false RuntimeLabels:map[app:k3d k3d.cluster:test k3d.cluster.imageVolume:k3d-test-images k3d.cluster.network:k3d-test k3d.cluster.network.external:false k3d.cluster.network.id:adaa120f35715aecf62ae28c14869e43b553c99c1625eb69ece8877fbb9fb0da k3d.cluster.network.iprange:10.89.1.0/24 k3d.cluster.token:VleGSYfkgIMqsIxqKDJU k3d.cluster.url:https://k3d-test-server-0:6443 k3d.server.loadbalancer:k3d-test-serverlb] RuntimeUlimits:[] K3sNodeLabels:map[] Networks:[k3d-test] ExtraHosts:[] ServerOpts:{IsInit:false KubeAPI:0x140003594c0} AgentOpts:{} GPURequest: Memory: State:{Running:false Status: Started:} IP:{IP:invalid IP Static:false} HookActions:[]} 
TRAC[0001] Creating docker container with translated config
&{ContainerConfig:{Hostname:k3d-test-server-0 Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[] Tty:false OpenStdin:false StdinOnce:false Env:[K3S_TOKEN=VleGSYfkgIMqsIxqKDJU K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml] Cmd:[server --tls-san 0.0.0.0 --tls-san k3d-test-serverlb] Healthcheck:<nil> ArgsEscaped:false Image:docker.io/rancher/k3s:v1.27.4-k3s1 Volumes:map[] WorkingDir: Entrypoint:[/bin/k3d-entrypoint.sh] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[app:k3d k3d.cluster:test k3d.cluster.imageVolume:k3d-test-images k3d.cluster.network:k3d-test k3d.cluster.network.external:false k3d.cluster.network.id:adaa120f35715aecf62ae28c14869e43b553c99c1625eb69ece8877fbb9fb0da k3d.cluster.network.iprange:10.89.1.0/24 k3d.cluster.token:VleGSYfkgIMqsIxqKDJU k3d.cluster.url:https://k3d-test-server-0:6443 k3d.role:server k3d.server.api.host:0.0.0.0 k3d.server.api.hostIP:0.0.0.0 k3d.server.api.port:57915 k3d.server.loadbalancer:k3d-test-serverlb k3d.version:v5.6.0] StopSignal: StopTimeout:<nil> Shell:[]} HostConfig:{Binds:[k3d-test-images:/k3d/images] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode:bridge PortBindings:map[] RestartPolicy:{Name:unless-stopped MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] ConsoleSize:[0 0] Annotations:map[] CapAdd:[] CapDrop:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:true PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[/run: /var/run:] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] MaskedPaths:[] ReadonlyPaths:[] Init:0x14000155f8a} NetworkingConfig:{EndpointsConfig:map[k3d-test:0x14000012240]}} 
DEBU[0001] Created container k3d-test-server-0 (ID: 83bce54b7487081c386b262af1884e28bfb53b42b8248a25aeeccbab6b183818) 
DEBU[0001] Created node 'k3d-test-server-0'             
INFO[0001] Creating LoadBalancer 'k3d-test-serverlb'    
TRAC[0001] Creating node from spec
&{Name:k3d-test-serverlb Role:loadbalancer Image:ghcr.io/k3d-io/k3d-proxy:5.6.0 Volumes:[k3d-test-images:/k3d/images] Env:[] Cmd:[] Args:[] Ports:map[6443:[{HostIP:0.0.0.0 HostPort:57915}]] Restart:true Created: HostPidMode:false RuntimeLabels:map[app:k3d k3d.cluster:test k3d.cluster.imageVolume:k3d-test-images k3d.cluster.network:k3d-test k3d.cluster.network.external:false k3d.cluster.network.id:adaa120f35715aecf62ae28c14869e43b553c99c1625eb69ece8877fbb9fb0da k3d.cluster.network.iprange:10.89.1.0/24 k3d.cluster.token:VleGSYfkgIMqsIxqKDJU k3d.cluster.url:https://k3d-test-server-0:6443 k3d.role:loadbalancer k3d.server.loadbalancer:k3d-test-serverlb k3d.version:v5.6.0] RuntimeUlimits:[] K3sNodeLabels:map[] Networks:[k3d-test] ExtraHosts:[] ServerOpts:{IsInit:false KubeAPI:<nil>} AgentOpts:{} GPURequest: Memory: State:{Running:false Status: Started:} IP:{IP:invalid IP Static:false} HookActions:[{Stage:preStart Action:{Runtime:{} Content:[112 111 114 116 115 58 10 32 32 54 52 52 51 46 116 99 112 58 10 32 32 45 32 107 51 100 45 116 101 115 116 45 115 101 114 118 101 114 45 48 10 115 101 116 116 105 110 103 115 58 10 32 32 119 111 114 107 101 114 67 111 110 110 101 99 116 105 111 110 115 58 32 49 48 50 52 10] Dest:/etc/confd/values.yaml Mode:-rwxr--r-- Description:Write Loadbalancer Configuration}}]} 
TRAC[0001] Creating docker container with translated config
&{ContainerConfig:{Hostname:k3d-test-serverlb Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[6443:{}] Tty:false OpenStdin:false StdinOnce:false Env:[K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml] Cmd:[] Healthcheck:<nil> ArgsEscaped:false Image:ghcr.io/k3d-io/k3d-proxy:5.6.0 Volumes:map[] WorkingDir: Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[app:k3d k3d.cluster:test k3d.cluster.imageVolume:k3d-test-images k3d.cluster.network:k3d-test k3d.cluster.network.external:false k3d.cluster.network.id:adaa120f35715aecf62ae28c14869e43b553c99c1625eb69ece8877fbb9fb0da k3d.cluster.network.iprange:10.89.1.0/24 k3d.cluster.token:VleGSYfkgIMqsIxqKDJU k3d.cluster.url:https://k3d-test-server-0:6443 k3d.role:loadbalancer k3d.server.loadbalancer:k3d-test-serverlb k3d.version:v5.6.0] StopSignal: StopTimeout:<nil> Shell:[]} HostConfig:{Binds:[k3d-test-images:/k3d/images] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode:bridge PortBindings:map[6443:[{HostIP:0.0.0.0 HostPort:57915}]] RestartPolicy:{Name:unless-stopped MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] ConsoleSize:[0 0] Annotations:map[] CapAdd:[] CapDrop:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:true PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[/run: /var/run:] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] MaskedPaths:[] ReadonlyPaths:[] Init:0x1400020db9a} NetworkingConfig:{EndpointsConfig:map[k3d-test:0x14000012300]}} 
DEBU[0001] Created container k3d-test-serverlb (ID: a3041bc1f0c6f5cb6c49d432c0a77226d543b2bf22d1e9866c35c68362255c44) 
DEBU[0001] Created loadbalancer 'k3d-test-serverlb'     
DEBU[0001] DOCKER_SOCK=/var/run/docker.sock             
INFO[0001] Using the k3d-tools node to gather environment information 
TRAC[0001] TranslateContainerDetailsToNode: Checking for default object label app=k3d on container /k3d-test-tools 
DEBU[0001] no netlabel present on container /k3d-test-tools 
DEBU[0001] failed to get IP for container /k3d-test-tools as we couldn't find the cluster network 
INFO[0001] Starting existing tools node k3d-test-tools... 
INFO[0001] Starting Node 'k3d-test-tools'               
ERRO[0001] failed to gather environment information used for cluster creation: error starting existing tools node k3d-test-tools: docker failed to start container for node 'k3d-test-tools': Error response from daemon: failed to create new hosts file: unable to replace "host-gateway" of host entry "host.k3d.internal:host-gateway": host containers internal IP address is empty 
FATA[0001] Cluster creation FAILED, rollback deactivated. 

Which OS & Architecture

k3d runtime-info
arch: arm64
cgroupdriver: systemd
cgroupversion: "2"
endpoint: /var/run/docker.sock
filesystem: xfs
infoname: localhost.localdomain
name: docker
os: fedora
ostype: linux
version: 4.7.0

Which version of k3d

k3d version
k3d version v5.6.0
k3s version v1.27.4-k3s1 (default)

Which version of docker

podman connection is rootful

podman version
Client:       Podman Engine
Version:      4.7.2
API Version:  4.7.2
Go Version:   go1.21.3
Git Commit:   750b4c3a7c31f6573350f0b3f1b787f26e0fe1e3
Built:        Tue Oct 31 14:59:17 2023
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      4.7.0
API Version:  4.7.0
Go Version:   go1.21.1
Built:        Wed Sep 27 21:17:40 2023
OS/Arch:      linux/arm64

Which version of OS

M1 MacOS 13.6

[eugene-chernyshenko]

Also I did this

[karlwowza]

Same problem on Mac M1 and Mac Intel in rootless and rootful mode.

Mac M1 details:

Client: Podman Engine Version: 4.8.2 API Version: 4.8.2 Go Version: go1.21.5 Git Commit: aa546902fa1a927b3d770528565627d1395b19f3 Built: Mon Dec 11 11:38:03 2023 OS/Arch: darwin/arm64

Server: Podman Engine Version: 4.7.2 API Version: 4.7.2 Go Version: go1.21.1 Built: Tue Oct 31 15:30:33 2023 OS/Arch: linux/arm64

and

k3d version v5.6.0 k3s

version v1.27.4-k3s1 (default)

[rstoermer]

Same problem here

[alewkinr]

The same, any resolution here?

[grozan]

I guess it's related to this podman issue: containers/podman#21681