[metricbeat] undocumented behavior of system/socket metricset when running in docker

[fearful-symmetry]

For confirmed bugs, please report:

• Version: 7.0
• Operating System: linux
• Steps to Reproduce:

1. setup docker based on the configuration here: https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-docker.html

docker run \
  --user=root \
  --name=metricbeat-test \
  --mount type=bind,source=/proc,target=/hostfs/proc,readonly \
  --mount type=bind,source=/sys/fs/cgroup,target=/hostfs/sys/fs/cgroup,readonly \
  --mount type=bind,source=/,target=/hostfs,readonly \
  --net=host \
  --volume="$(pwd)/metricbeat.docker.yml:/usr/share/metricbeat/metricbeat.yml:ro" \
   docker.elastic.co/beats/metricbeat:7.0.0 \
  -e -system.hostfs=/hostfs

2. enable the system/socket metricset

3. Get some weird errors:

  "system": {
    "socket": {
      "local": {
        "port": 22,
        "ip": "::"
      }
    }
  },
  "ecs": {
    "version": "1.0.0"
  },
  "server": {
    "ip": "::",
    "port": 22
  },
  "error": {
    "code": "process not found. inode=29023, tcp_state=LISTEN"
  },

What's happening here is that the metricset is using netlink to actually grab socket data, and then using procfs to map the inodes back to processes. Even when running as root the container process doesn't have permissions to access the host symlinks in /proc/$pid/fd, hence the error. A temporary workaround is to use --privileged, which is a tad blunt. You can also omit the --user=root, although having metricbeat just monitor its own socket usage isn't that helpful, as an info message will tell you:

INFO	socket/socket.go:81	socket process info will only be available for metricbeat because the process is running as a non-root user

It seems to be we should either alter the behavior of the metricset or document how to work around this. Having metricbeat monitor the host system from inside docker seems to be a supported use case, and I don't see anything on the above-linked page that would allow the socket metricset to just work. We may also want to document that without --user=root the metricset will just monitor itself, as --user=root is mentioned on the above-linked page, but never really explained. We may also want to do some kind of permissions check at runtime and print another info message similar to the one above.