DNS resolution failing on EKS slaves

[andyhopp]

Summary

DNS resolution appears to be broken when running a docker build on EKS. I have followed the Kubernetes DNS resolution troubleshooting guide at https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/, and I added logging to CoreDNS. While I see the queries for the container images used by the Dockerfile, I do not see any of the queries that should be issued from running the commands in the Dockerfile.

Steps to reproduce the behavior

1. Create an EKS Jenkins X cluster following instructions at https://aws.amazon.com/blogs/opensource/continuous-delivery-eks-jenkins-x/
2. Install the aspnet quick start: jx create quickstart, choosing the "aspnet-app" sample and follow prompts for GitHub orgs, etc.

Expected behavior

Build proceeds without errors.

Actual behavior

Build fails with the below in the log:

  Restoring packages for /app/aspnetapp.csproj...

  Restoring packages for /app/aspnetapp.csproj...

/usr/share/dotnet/sdk/2.1.202/NuGet.targets(114,5): error : Unable to load the service index for source https://api.nuget.org/v3/index.json. [/app/aspnetapp.csproj]

/usr/share/dotnet/sdk/2.1.202/NuGet.targets(114,5): error :   An error occurred while sending the request. [/app/aspnetapp.csproj]

/usr/share/dotnet/sdk/2.1.202/NuGet.targets(114,5): error :   Couldn't resolve host name [/app/aspnetapp.csproj]

time="2019-09-12T15:56:56Z" level=fatal msg="build failed: build failed: building [andyhopp/aspnet-app]: build artifact: unable to stream build output: The command '/bin/sh -c dotnet restore' returned a non-zero code: 1"

Jx version

The output of jx version is:

NAME               VERSION
jx                 2.0.712
jenkins x platform 2.0.1165
Kubernetes cluster v1.13.10-eks-5ac0f1
kubectl            v1.14.6
helm client        Client: v2.14.2+ga8b13cc
git                2.17.2 (Apple Git-113)
Operating System   Mac OS X 10.13.6 build 17G8030

Jenkins type

• Serverless Jenkins X Pipelines (Tekton + Prow)
• Classic Jenkins

Kubernetes cluster

EKS created using guide linked above

Operating system / Environment

MacOS

[andyhopp]

I did find that resolv.conf contains this, which does not appear correct:

; generated by /usr/sbin/dhclient-script
search ec2.internal
options timeout:2 attempts:5
nameserver 192.168.0.2

I would have expected something like (this was on a pod I manually launched):

nameserver 10.100.0.10
search jx.svc.cluster.local svc.cluster.local cluster.local ec2.internal
options ndots:5

[andyhopp]

I discovered the issue; we disabled the docker bridge in EKS (see awslabs/amazon-eks-ami#183). Updating skaffold.yaml to configure docker builds to use host networking resolved this:

...
build:
  artifacts:
    ...
    docker:
      network: host

[abdennour]

Found the answer : https://stackoverflow.com/a/57934460/747579

Just add --network=host to docker build or docker run

[deanesmith]

@andyhopp We'll be supporting Kaniko images very soon but it appears a Docker workaround is there with @abdennour's comment. Ref - #5433. Please close this issue if you're satisfied. P.S. - Remember Ipswitch?

[andyhopp]

Ha, I realized it was you on the other issue I had opened! I'll hit you up outside this issue so I don't annoy people who are trying to solve problems :)

[DJEDAINI]

I discovered the issue; we disabled the docker bridge in EKS (see awslabs/amazon-eks-ami#183). Updating skaffold.yaml to configure docker builds to use host networking resolved this:

...
build:
  artifacts:
    ...
    docker:
      network: host

@andyhopp

You saved my day man , thank you