Dial tcp: lookup portainer_agent, no such host when using environment with agent

[SirMishaa]

Bug description
Hello, I have just setup portainer with a new Docker installation (On Ubuntu 23.04 Server distribution), the problem is that Portainer seems unable to communicate with the agent, even though it is running fine.

Expected behavior
A clear and concise description of what you expected to happen.

Portainer Logs

mishaa@ubuntu ~> docker ps
CONTAINER ID   IMAGE                           COMMAND        CREATED         STATUS         PORTS                                                                                                                             NAMES
98b529ac7956   portainer/agent:2.18.3-alpine   "./agent"      5 minutes ago   Up 5 minutes   0.0.0.0:9001->9001/tcp, :::9001->9001/tcp                                                                                         portainer_agent
4d903b8ba50a   portainer/portainer-ee:latest   "/portainer"   8 minutes ago   Up 8 minutes   0.0.0.0:8000->8000/tcp, :::8000->8000/tcp, 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp, 0.0.0.0:9443->9443/tcp, :::9443->9443/tcp   portainer

mishaa@ubuntu ~> docker logs 98b529ac7956fb147a494dafcc52a2a1dd40ede2cdec713b7446e82921aeb5fb
2023/06/12 12:17AM INF github.com/portainer/agent/cmd/agent/main.go:86 > agent running on Docker platform |
2023/06/12 12:17AM INF github.com/portainer/agent/edge/registry/server.go:101 > starting registry credential server |
2023/06/12 12:17AM INF github.com/portainer/agent/http/server.go:99 > starting Agent API server | api_version=2.18.3 server_addr=0.0.0.0 server_port=9001 use_tls=true

mishaa@ubuntu ~> docker logs 4d903b8ba50a3ffd448f2c4dd433a493d97d91798965e90c296350031e1504bf
2023/06/12 12:14AM INF main.go:568 > encryption key file not present | filename=portainer
2023/06/12 12:14AM INF main.go:602 > proceeding without encryption key |
2023/06/12 12:14AM INF github.com/portainer/portainer-ee/api/database/boltdb/db.go:124 > loading PortainerDB | filename=portainer.db
2023/06/12 12:14AM INF github.com/portainer/portainer-ee/api/cloud/cluster_setup.go:93 > starting cloud cluster setup service |
2023/06/12 00:14:52 server: Reverse tunnelling enabled
2023/06/12 00:14:52 server: Fingerprint b7:37:bc:df:70:d6:f5:b2:75:4a:f5:09:fb:f7:20:fe
2023/06/12 00:14:52 server: Listening on 0.0.0.0:8000...
2023/06/12 12:14AM INF main.go:902 > starting Portainer | build_number=30955 go_version=1.19.4 image_tag=linux-arm64-2.18.3 nodejs_version=18.16.0 version=2.18.3 webpack_version=5.65.0 yarn_version=1.22.19
2023/06/12 12:14AM INF github.com/portainer/portainer-ee/api/http/server.go:402 > starting HTTP server | bind_address=:9000
2023/06/12 12:14AM INF github.com/portainer/portainer-ee/api/http/server.go:417 > starting HTTPS server | bind_address=:9443

mishaa@ubuntu ~> sudo lsof -i -P -n | grep LISTEN
systemd       1            root   67u  IPv6  18076      0t0  TCP *:22 (LISTEN)
systemd-r   484 systemd-resolve   14u  IPv4  17102      0t0  TCP 127.0.0.53:53 (LISTEN)
systemd-r   484 systemd-resolve   16u  IPv4  17104      0t0  TCP 127.0.0.54:53 (LISTEN)
sshd       1564            root    3u  IPv6  18076      0t0  TCP *:22 (LISTEN)
docker-pr 14409            root    4u  IPv4  74465      0t0  TCP *:9443 (LISTEN)
docker-pr 14416            root    4u  IPv6  74472      0t0  TCP *:9443 (LISTEN)
docker-pr 14430            root    4u  IPv4  76777      0t0  TCP *:9000 (LISTEN)
docker-pr 14438            root    4u  IPv6  73517      0t0  TCP *:9000 (LISTEN)
docker-pr 14454            root    4u  IPv4  74484      0t0  TCP *:8000 (LISTEN)
docker-pr 14462            root    4u  IPv6  74487      0t0  TCP *:8000 (LISTEN)
docker-pr 14806            root    4u  IPv4  75751      0t0  TCP *:9001 (LISTEN)
docker-pr 14814            root    4u  IPv6  75754      0t0  TCP *:9001 (LISTEN)

Technical details:

• Portainer version: Latest (just pulled right now)
• Docker version (managed by Portainer): Docker version 24.0.2, build cb74dfc
• Kubernetes version (managed by Portainer): /
• Platform (windows/linux): Ubuntu 23.04 Server
• Command used to start Portainer (docker run -p 9443:9443 portainer/portainer):

docker run -d -p 8000:8000 -p 9443:9443 --name=portainer --restart=always \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v portainer_data:/data \
    portainer/portainer-ee:latest

Agent

docker run -d \
  -p 9001:9001 \
  --name portainer_agent \
  --restart=always \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /var/lib/docker/volumes:/var/lib/docker/volumes \
  portainer/agent:2.18.3

• Use Case (delete as appropriate): Using Portainer at Home, Using Portainer in a Commercial setup.
• Have you reviewed our technical documentation and knowledge base? Yes

Additional context
Add any other context about the problem here.

I just followed the documentation

[SirMishaa]

It looks like creating a network solving the issue. By default, the default network is on bridge ? Could I have some issue with my newly created network ?

We don't say that we need to create a network in the documentation.

mishaa@ubuntu ~> docker network ls
NETWORK ID     NAME      DRIVER    SCOPE
fadff2557276   bridge    bridge    local
af59fe38a184   host      host      local
cde627748b44   none      null      local
mishaa@ubuntu ~> docker network create portainer
2a20a53f29c8700f8dff9f9f712d32c0170b5e96054f69bb40c91871f1384261
mishaa@ubuntu ~> docker network connect 2a20a53f29c8 98
mishaa@ubuntu ~> docker network connect 2a20a53f29c8 4d

mishaa@ubuntu ~ [1]> docker network ls
NETWORK ID     NAME        DRIVER    SCOPE
fadff2557276   bridge      bridge    local
af59fe38a184   host        host      local
cde627748b44   none        null      local
2a20a53f29c8   portainer   bridge    local

[jamescarppe]

Are you running the Portainer Server and Portainer Agent on the same machine? You don't need to do this - the Portainer Server installation lets you manage your local environment without needing the agent. There is some additional functionality that an agent gives you over the plain server installation (host file system browsing for example) but for most things you shouldn't need to do this.

It also looks like your environment address isn't right - you're trying to use portainer_agent as the address where this should be the IP or hostname of the server you're connecting to. In the case of your local server, you could try localhost or 127.0.0.1, or the actual IP of the server.

The reason creating a network "solved" the issue is that the default bridge network doesn't support DNS resolution within the network, which is why the hostname of portainer_agent didn't work. Manually created bridge networks do support DNS resolution. But in any case, you shouldn't need to use this approach for the reasons above.