can we use NLB with EKS clusters created via eksctl?

[jstrachan]

I've been trying out NLB instead of ELB as NLBs have stable static IP addresses; so you can use the nip.io trick of setting up Ingress without mandating a custom DNS name being used to setup wildcard CND via a CNAME to kick the tyres on EKS.

Things work great on kops but don't seem to work on EKS which am guessing is related to roles. It looks like these kinds of roles are required for setting up NLB on AWS with kops
https://gist.github.com/micahhausler/4f3a2ee540f5714e6dd91b4bacace3ae#file-create-cluster-sh-L30

so am guessing something similar is required on the NodeInstanceRole. Will try figure out whats really required...

[errordeveloper]

You probably try adding those at cluster level:

https://github.com/weaveworks/eksctl/blob/c332060348833792c9a69f1ef71d1c8d76c3ffd7/pkg/eks/assets/1.10.3/2018-06-05/amazon-eks-service-role.yaml#L8-L22

[errordeveloper]

FYI, I'm in a middle of refactoring these things (see #127 and #132), but I'm not sure yet if we will land refactoring changes before 0.1.0 or after.

[jstrachan]

@errordeveloper thanks for the heads up.

Not sure if this is related to roles on the nodes or not; tried manually adding the nodes and no luck - can't seem to get the NLB working at all on EKS via eksctl. No clue why its not working TBH - this all just works on kops when those roles are added - so no idea if there's anything we can do in eksct to help get NLB working with ingress.

I wonder if you're right, we need the role added to the cluster/EKS control plane too?

[jstrachan]

BTW I've been following these instructions to setup NLB on the EKS cluster and no luck so far when the same thing works fine on kops:
https://aws.amazon.com/blogs/opensource/network-load-balancer-support-in-kubernetes-1-9/

[jstrachan]

btw after chatting with @errordeveloper on slack - this issue could be that NLB cannot be used with EKS due to the beta annotation - still not 100% sure yet...

[errordeveloper]

cc @d-nishi @nckturner

[jstrachan]

aha thanks to this tweet and noodling those related docs I found that adding this policy to the control plane role worked!

{
   "Version": "2012-10-17",
   "Statement": [{
      "Effect": "Allow",
      "Action": [
          "elasticloadbalancing:*",
          "ec2:CreateSecurityGroup",
          "ec2:Describe*"
      ],
      "Resource": "*"
    }
   ]
}

[errordeveloper]

Great! I am happy to enable this by default, seems useful for everyone. Wearing for a PR!

…

On Tue, 24 Jul 2018, 1:28 pm James Strachan, ***@***.***> wrote: aha thanks to this tweet <https://twitter.com/micahhausler/status/1021563940763885569> and noodling those related docs I found that adding this policy to the control plane role worked! { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": [ "elasticloadbalancing:*", "ec2:CreateSecurityGroup", "ec2:Describe*" ], "Resource": "*" } ] } — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#133 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAPWS7Gc0lrxM7dFLL8e_nq_Hupxl2IUks5uJxL6gaJpZM4Vas2j> .

[jstrachan]

@errordeveloper great thanks! Was just gonna wait for the CloudFormation PRs to settle so I can see how best to do it; as using CloudFormation for the control plane is maybe a nice idea?

[errordeveloper]

@jstrachan I don't think #132 will land before next week, in fact I'd rather not land it before 0.1.0.

[errordeveloper]

I will add this to 0.1.0 milestone, in a way it's a bug too.

[aparamon]

Hmm, I have annotated my service with service.beta.kubernetes.io/aws-load-balancer-type=nlb, but I still get a ELB, not NLB attached:

>kubectl describe svc genghis
Name:                     genghis
Namespace:                default
Labels:                   <none>
Annotations:              kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"s
ervice.beta.kubernetes.io/aws-load-balancer-type":"nlb"},"name":"genghis","namespace":...
                          service.beta.kubernetes.io/aws-load-balancer-type=nlb
Selector:                 app=genghis
Type:                     LoadBalancer
IP:                       10.100.135.56
LoadBalancer Ingress:     a7f89af5f249711e9a0160683d1063c6-892f351b073a621b.elb.eu-central-1.amazonaws.com
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  30609/TCP
Endpoints:                192.168.114.45:80,192.168.131.233:80,192.168.159.254:80 + 1 more...
Session Affinity:         None
External Traffic Policy:  Cluster

Do I need to create the NLB beforehand?

[errordeveloper]

@aparamon no, you don't need to create it beforehand. Did you manage to troubleshoot this? If not, please open another issue - we don't normally monitor closed issues.

[aparamon]

@errordeveloper I didn't, but in the end I used different architecture with a dedicated load-balancer node which runs ingress controller only and possesses static external IP (Elastic IP).
In fact, I realized that level 4 load-balancer (NLB) doesn't suit my application anyway. So EKS was somehow doing the right thing ;-)