Loki asking DNS to resolve localhost

[SerialVelocity]

Describe the bug
When monitoring my DNS server (to reduce the amount of DNS calls because of Kubernetes ndots:5), I noticed that Loki is asking the DNS server to resolve localhost every minute:

loki-0                         2020-04-10T22:38:49.597522713Z [INFO] 172.30.23.68:44644 - 58469 "AAAA IN localhost.monitoring.svc.k8s.internal. udp 62 false 512" NXDOMAIN qr,aa,rd 173 0.004305404s
loki-0                         2020-04-10T22:38:49.598313764Z [INFO] 172.30.23.68:36740 - 33883 "A IN localhost.monitoring.svc.k8s.internal. udp 62 false 512" NXDOMAIN qr,aa,rd 173 0.005249183s
loki-0                         2020-04-10T22:38:49.609921919Z [INFO] 172.30.23.68:54820 - 37430 "AAAA IN localhost.svc.k8s.internal. udp 51 false 512" NXDOMAIN qr,aa,rd 162 0.000563601s
loki-0                         2020-04-10T22:38:49.609983283Z [INFO] 172.30.23.68:50490 - 52082 "A IN localhost.svc.k8s.internal. udp 51 false 512" NXDOMAIN qr,aa,rd 162 0.000255911s
loki-0                         2020-04-10T22:38:49.622573997Z [INFO] 172.30.23.68:54596 - 17609 "A IN localhost.k8s.internal. udp 51 false 512" NXDOMAIN qr,aa,rd 158 0.000545007s
loki-0                         2020-04-10T22:38:49.622645462Z [INFO] 172.30.23.68:55064 - 15223 "AAAA IN localhost.k8s.internal. udp 51 false 512" NXDOMAIN qr,aa,rd 158 0.000380376s
loki-0                         2020-04-10T22:38:49.63635297Z  [INFO] 172.30.23.68:43947 - 34659 "A IN localhost. udp 27 false 512" NXDOMAIN qr,rd,ra 102 0.014425104s
loki-0                         2020-04-10T22:38:49.655885416Z [INFO] 172.30.23.68:35308 - 25243 "AAAA IN localhost. udp 27 false 512" NXDOMAIN qr,rd,ra 102 0.018790019s

To Reproduce
Steps to reproduce the behaviour:

1. Start Loki v1.3.0 or 1.4.1
2. (I had data being ingested, not sure if that matters
3. Watch DNS logs

Expected behaviour
A clear and concise description of what you expected to happen.

Environment:

• Infrastructure: Kubernetes 1.17.4 with containerd and coredns
• Deployment tool: yaml

Screenshots, Promtail config, or terminal output
Config:

    auth_enabled: false
    server:
      http_listen_port: 3100
    ingester:
      chunk_idle_period: 3m
      chunk_block_size: 262144
      chunk_retain_period: 1m
      max_transfer_retries: 0
      lifecycler:
        ring:
          kvstore:
            store: inmemory
          replication_factor: 1
    limits_config:
      enforce_metric_name: false
      reject_old_samples: true
      reject_old_samples_max_age: 168h
    schema_config:
      configs:
      - from: 2018-04-15
        store: boltdb
        object_store: filesystem
        schema: v9
        index:
          prefix: index_
          period: 168h
    storage_config:
      boltdb:
        directory: /data/loki/index
      filesystem:
        directory: /data/loki/chunks
    chunk_store_config:
      max_look_back_period: 0
    table_manager:
      retention_deletes_enabled: false
      retention_period: 0

[SerialVelocity]

Found this alpine issue about nsswitch.conf missing:
gliderlabs/docker-alpine#367

Which is actually caused by Go having unexpected behaviour when /etc/nsswitch.conf is missing:
golang/go#35305

It looks like most people are adding a valid /etc/nsswitch.conf to workaround this issue until Go fixes it:

RUN [ ! -e /etc/nsswitch.conf ] && echo 'hosts: files dns' > /etc/nsswitch.conf

[slim-bean]

This behavior is likely intentional, Loki running as a single binary actually makes GRPC calls to itself, I'm wondering if in doing so it periodically makes DNS requests to resolve localhost.

Thanks for doing the legwork on that alpine issue. Is your understanding that if we add that workaround the DNS calls would be handled by the etc/hosts file and thus prevent the need for an external DNS call?

[SerialVelocity]

@slim-bean Yes, I just tested it and it stopped the DNS calls.

I threw up #1949 to fix this and also cleanup the Dockerfile a bit

[SerialVelocity]

prevent the need for an external DNS call?

Also, DNS servers don't always return a response for localhost so you actually get NXDOMAIN back (CoreDNS does this). After the 8 DNS calls have taken place, Go will finally check /etc/hosts and find it in there.