[Pi-hole] Only a single client

[marcelstoer]

Since Pi-hole runs in a Docker container on the default Docker network, it identifies all clients in its logs as 172.X.0.1.

This is discussed in a number of places e.g. https://discourse.pi-hole.net/t/pi-hole-in-docker-on-macos-has-only-one-client/55875

[Paraphraser]

Well, that's not what I see. For example:

To be honest, I can't understand how Pi-hole would ever actually see 172.X.0.1 in incoming query packets.

I'm assuming that 172.X.0.1 implies the default gateway on the internal bridged network.

Please see if you can spot a hole (no pun intended) in my reasoning. Make these assumptions:

1. Pi-hole is running in a non-host-mode Docker container on a Raspberry Pi.
2. Docker assigns 172.16.0.4/16 to the Pi-hole container.
3. The Raspberry Pi's IP address is 192.168.0.60/24.
4. A client has the IP address 192.168.0.100/24.

When the client issues a DNS query, the source IP is 192.168.0.100 and the destination IP is 192.168.0.60.

When the packet arrives at the Pi, iptables rules set up by Docker rewrite (de-NAT) the destination address to be 172.16.0.4, then the packet is routed to Pi-hole.

The source IP isn't changed so Pi-hole can see 192.168.0.100 and can log that against the query.

The reply to the query will have a source IP of 172.16.0.4 and a destination IP of 192.168.0.100.

The packet routes out of the internal bridged network to the Pi where iptables rules rewrite (NAT) the source IP address to be 192.168.0.60, and the packet is then forwarded back to the client.

It's fairly easy to verify this:

$ docker exec -it pihole bash
# apt update
# apt install -y tcpdump
# tcpdump -n port 53

I have yet to spot the IP address of the default gateway on the internal bridged network (ie 172.16.0.1). All incoming queries originating from devices on my home network have source IP addresses matching 192.168.132.x.

Outside container-space, you can also use tcpdump to confirm that addresses from the internal bridged network aren't getting out onto your home network:

$ sudo tcpdump -n port 53

I see plenty of queries and replies, both answered directly by Pi-hole, or being relayed to BIND9 or 8.8.8.8 etc, but no sign of 172.16.x.x.

---

It's possible that the reason I see all my local clients is because of how I have Pi-hole configured so here's my service definition:

  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    restart: unless-stopped
    environment:
      - TZ=${TZ:-Etc/UTC}
      - WEBPASSWORD=
      - INTERFACE=eth0
      - PIHOLE_DNS_=8.8.8.8;8.8.4.4
      - REV_SERVER=true
      - REV_SERVER_DOMAIN=my.domain.com
      - REV_SERVER_TARGET=192.168.132.55
      - REV_SERVER_CIDR=192.168.132.0/24
      - FTLCONF_MAXDBDAYS=2
    ports:
      - "53:53/tcp"
      - "53:53/udp"
      - "67:67/udp"
      - "8089:80/tcp"
    volumes:
      - ./volumes/pihole/etc-pihole:/etc/pihole
      - ./volumes/pihole/etc-dnsmasq.d:/etc/dnsmasq.d
    dns:
      - 127.0.0.1
      - 1.1.1.1
    cap_add:
      - NET_ADMIN

The way I use Pi-hole is like this. DHCP divvies-up clients into two buckets:

• devices that need ad-blocking services, which are sent to Pi-hole; and
• everything else, which are sent to my local BIND9 instance (192.168.132.55).

BIND9 is authoritative for my.domain.com. and 132.168.192.in-addr.arpa.

When a client directs a query to Pi-hole, there are three possible outcomes:

1. the query is blocked (ie matches an ad-list); or
2. the query is in a local domain (forward or reverse), in which case it will be relayed to the local BIND9 instance; otherwise
3. the query will be relayed to 8.8.8.8 and friends.

Because BIND9 can answer reverse queries, Pi-hole can translate local IP addresses to local domain names, which then show up in the GUI (the example above).

---

From the Discourse post:

where X seems to change after a reboot

The "problem" of Docker allocating the subnet dynamically when it creates the default network can be "solved" like this:

networks:

  default:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 172.30.0.0/22

Fairly obviously you can pick any subnet you like and prefix length you like. Some people would rather let Docker make the choice (which is why I put "problem" in quotes) but I prefer predictability.

[Paraphraser]

By the way, the 10.244.124.118 in the client list is a remote device communicating via ZeroTier Cloud. My BIND9 instance does define a domain name for that but Pi-hole can't reverse resolve the IP to the name because the REV_SERVER_CIDR variable and the corresponding field in the Pi-hole GUI only supports a single range.

[marcelstoer]

Thank you Phill for taking the time to give such a thorough answer! At the same time I must apologize for my low-quality issue report; a bit embarrassing really.

For one, I did not mention that my IOTstack runs on a Mac. Your feedback and that of the author of the issue I linked to, seem to confirm this be no issue on the Pi ("running Pi-hole in a Raspberry Pi...without any issues").

So, I guess other than maybe adding a hint IOTstack Pi-hole documentation for Mac users nothing needs changing.

[Paraphraser]

Ah! Gotcha. My bad.

But there's no need for any apologies, mate. Any question that results in new knowledge is worthwhile.

TL;DR

I agree that:

1. The observed behaviour for Pi-hole running in Docker Desktop on macOS is to log all clients as the default gateway on the iotstack_default network; and
2. It would be a good idea to document this as "expected behaviour".

Now the TL bit

I did run Docker Desktop for Mac for a while but the darn thing kept giving me the screaming rhymes-with-fits. It would either not start, or not shut down cleanly, or crash, or would threaten me that this, that or the other would break if I didn't stand on one leg and whistle Waltzing Matilda in E minor. Eventually, I nuked it. That was about a year ago.

When I was running Docker Desktop for Mac, I mostly stuck with MING. I'm pretty sure I never tried any containers that needed privileged ports (eg Pi-hole) because it was my understanding it wouldn't work.

So, read all that as some limited experience with the environment, but nothing directly related to the problem at hand.

That said, I just installed it on my M2 MacBookPro.

$ docker exec -it pihole bash
# apt update
# apt install -y traceroute tcpdump net-tools

I agree that PiHole records the IP address of the default gateway for all clients. In the following, the querying host is 192.168.132.60 but, in the first line of tcpdump output, that source IP has indeed been replaced with the IP address of the default gateway on the iotstack_default network; and that's where Pi-hole returns the reply (4^th line):

# tcpdump -c 4 -tn port 53
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
IP 172.30.0.1.39948 > 172.30.0.2.53: 50406+ [1au] A? www.apple.com. (54)
IP 172.30.0.2.46240 > 8.8.4.4.53: 3904+ [1au] A? www.apple.com. (54)
IP 8.8.4.4.53 > 172.30.0.2.46240: 3904 4/0/1 CNAME www.apple.com.edgekey.net., CNAME www.apple.com.edgekey.net.globalredir.akadns.net., CNAME e6858.dscx.akamaiedge.net., A 23.1.23.66 (192)
IP 172.30.0.2.53 > 172.30.0.1.39948: 50406 4/0/1 CNAME www.apple.com.edgekey.net., CNAME www.apple.com.edgekey.net.globalredir.akadns.net., CNAME e6858.dscx.akamaiedge.net., A 23.1.23.66 (192)

I think I see why. This is from inside the container:

# traceroute -nI 192.168.132.60 
traceroute to 192.168.132.60 (192.168.132.60), 30 hops max, 60 byte packets
 1  172.30.0.1  0.218 ms  0.030 ms  0.022 ms
 2  192.168.65.5  0.096 ms  0.034 ms  0.031 ms
 3  192.168.132.60  7.999 ms  8.251 ms  11.718 ms

The traceroute back to the querying host has three hops. On a Pi, there are only two hops.

The 192.168.65.0/24 subnet shows up in the Docker Desktop Settings»Resources»Network as "the resources network". Now there's a phrase to launch a thousand IP datagrams! My guess is that it represents another level of NAT. How the .5 fits in is a mystery.

Earlier, I said I had believed that privileged ports (anything less than 1024) wouldn't work. Because I never tried it before nuking everything, I can't say whether it was true then but this additional level of NAT has come along since as the fix, or if this extra level of NAT was always present, there never was any restriction on containers using privileged ports, and my belief was misplaced.

But this whole thing is really weird. Inside Pi-hole:

# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         172.30.0.1      0.0.0.0         UG        0 0          0 eth0
172.30.0.0      0.0.0.0         255.255.252.0   U         0 0          0 eth0

No sign of anything other than the iotstack_default network. From inside the container, you can ping the default gateway (172.30.0.1), the bridge network (172.17.0.1), the mysterious 192.168.65.5 and the .1 on that subnet, the host Mac, and anything beyond. Other than 172.30.0.1 which is a local interface, all the others are reachable because non-local packets will follow the container's default route.

If you've ever tried to run netstat on macOS you'll know you get pages of crud so I won't fill up this post and expect you to wade through it. I'll just report my observations:

1. The Mac has no route to the bridge network (172.17/16 via the docker0 interface). You can see it in:

   $ docker network inspect bridge
   

   so it's "there". Just not in the routing table so it isn't reachable from the Mac, meaning you can't ping it from Terminal.

2. There is no route to the iotstack_default network (eg 172.30/22 via a br-xxxx interface). It's the same as the bridge network: you can inspect it but it isn't reachable. You can't ping the .1 (default gateway) or the Pi-hole container (.2 in this example).

3. There is no route to anything that so much as smells like 192.168.65.0/24, so it isn't reachable.

4. Using route get on macOS for any of the Docker networks returns the local physical interfaces and the IP address of my home router (ie it's following the Mac's default route).

   On the Pi, the equivalent command is ip route get and, aside from 192.168.65.0/24 which doesn't exist, answers point to Dockerspace interfaces.

All in all, some seriously hinky things are going on here. When I searched for the PID associated with port 53, it arrived at:

/Applications/Docker.app/Contents/MacOS/com.docker.backend -watchdog -native-api

The logical conclusion is that all this jiggery-pokery is going on inside Docker Desktop and it's all completely independent of macOS networking.

I. Hate. Opaque.

If this were a Pi, you could work around extra NAT shenanigans by placing the container into "host mode". But that doesn't work for containers in Docker Desktop for Mac. You can enable host mode. The containers come up. Neither Docker nor the containers report any errors or warnings. But the containers just aren't reachable.

I. Hate. Silent. Fails.

There's enough of that with Gatekeeper. Example. Until you add cron to the full disk access list, any cron job that goes near, say, ~/Documents just sees an empty folder. You think your jobs are working and there's no hint of anything amiss in your logs. They just don't work.

Have you come across repairHomePermissions yet? Photos wouldn't upload images from my iPhone. Just stalled. No errors. No warnings. Just no progress. Needs a trip to recovery mode to run that command, after which all fixed. Aarrgghhh!!

[marcelstoer]

I ran through the same set of commands as you did (thanks for those) and I see mostly identical results - but not 100%.

❯ docker exec -ti pihole bash
root@5708c0f2c278:/# hostname -i 
172.18.0.4
root@5708c0f2c278:/# tcpdump -c 4 -tn port 53
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
IP 172.18.0.1.64017 > 172.18.0.4.53: 62740+ Type65? fp2e7a.wpc.phicdn.net. (39)
IP 172.18.0.4.39947 > 9.9.9.9.53: 47945+ Type65? fp2e7a.wpc.phicdn.net. (39)
IP 9.9.9.9.53 > 172.18.0.4.39947: 47945 0/1/0 (95)
IP 172.18.0.4.53 > 172.18.0.1.64017: 62740 0/1/0 (95)
4 packets captured
4 packets received by filter
0 packets dropped by kernel
root@5708c0f2c278:/# traceroute -nI 192.168.0.36
traceroute to 192.168.0.36 (192.168.0.36), 30 hops max, 60 byte packets
 1  172.18.0.1  0.097 ms  0.067 ms  0.034 ms
 2  192.168.0.36  3.797 ms  3.890 ms  4.410 ms
root@5708c0f2c278:/# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         172.18.0.1      0.0.0.0         UG        0 0          0 eth0
172.18.0.0      0.0.0.0         255.255.0.0     U         0 0          0 eth0
root@5708c0f2c278:/# exit
exit
❯ lsof -i tcp:53
COMMAND    PID       USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
com.docke 1104 gatekeeper  151u  IPv6 0x2c2279402f6234f9      0t0  TCP *:domain (LISTEN)
❯ ps -ef 1104
  UID   PID  PPID   C STIME   TTY           TIME CMD
  501  1104  1101   0 Sat05PM ??        26:45.01 /Applications/Docker.app/Contents/MacOS/com.docker.backend -watchdog -native-api

-> The traceroute back to the querying host 192.168.0.36 has two(!) hops.

Without understanding that in detail I conclude that the behavior on Mac (and most likely Windows as well) is due to the fact that Docker doesn't run "natively" on those platforms but in a Linux VM instead.

[Paraphraser]

That traceroute is interesting. So, my tests were on an M2 MacBookPro with Docker Desktop freshly download yesterday, a clean clone of IOTstack, and the compose file fragments borrowed from the Pi.

I haven't repeated the test on my Intel iMac. Are you doing this on Intel or Apple silicon?

[marcelstoer]

In terms of hardware/software versions our two setups are very different. I run the IOTstack on a late 2012 Mac Mini, Catalina, Docker Desktop 4.15.

[Paraphraser]

Hmmm. Well, mine says version 4.20.1. I've just repeated everything on my Intel iMac and I see three hops.

I could just - barely - make sense of the observed behaviour (Pi-hole logging all clients as the default gateway) if I hypothesised extra NAT. And, based on that assumption, the 192.168.65.5 made sense as the vehicle for NAT.

But now ... ? 🤷‍♂️

---

To answer your earlier (implied) question, traceroute works by issuing packets, starting with a time-to-live (aka "hop count") of 1, with each successive packet incrementing the initial hop count. As packets arrive at devices, the hop count is decremented. If the count goes zero, the packet is returned to sender ("bounced"). It's the bounces that constitute the output. The actual idea of the time-to-live is to break forwarding loops when routers are misconfigured. Traceroute just takes advantage of that behaviour.

A device receiving a packet is either the intended recipient, or is expected to be configured as a router. If neither, it's an error and the packet is dropped as undeliverable. If it's a router, the packet will be forwarded to the next hop according to the device's routing table (in most cases that means following the default route).

So, any line in a traceroute, other than the last, tells you a couple of things. It tells you "here be a router". It also tells you the near side (aka ingress side) IP address of that router - but not the far side (egress side) IP address.

Plenty can go wrong with traceroutes because a lot of sites think they're improving security by blocking the packets, so you often see a lot of holes in the output. Parallel paths can also confuse traceroute a bit, particularly if two paths to a destination involve different hop-counts.

[marcelstoer]

No question about traceroute but thanks anyway 😄 My old Docker version is dictated by the old hardware: old Mini -> old OS -> old Docker.

[marcelstoer]

Just in case anyone follows this all the way down here, here's a related Pi-hole issue: pi-hole/docker-pi-hole#135 (goes back to 2017).

[Paraphraser]

And, on this same theme (people making it all the way to the end of this issue), the TL;DR bit of my post from June 2023 said it would be a good idea to document the behaviour. I submitted a PR to do that. This is what you see here: