Support Horizontal Scaling

[kate-osborn]

Proposed changes

Problem: NKG cannot be scaled horizontally because all replicas will write statuses to the Gateway API resources.

Solution: Add leader election to the status updater so that only one replica of NKG will write statuses to the Gateway API resources. Leader election is enabled by default but can be disabled via a cli arg --leader-election-disable. The lock name used for leader election can be configured via the cli arg --leader-election-lock-name.

Testing: Verified leader election works with multiple replicas.
Closes #637

Checklist

Before creating a PR, run through this checklist and mark each as complete.

• I have read the CONTRIBUTING doc
• I have added tests that prove my fix is effective or that my feature works
• I have checked that all unit tests pass after adding my changes
• I have updated necessary documentation
• I have rebased my branch onto main
• I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

[pleshakov]

more comments:
(1) Would it make sense to add something about leader election to the architecture doc?

(2) What removes the leader election lock resource? Do we need to update our deinstallation instructions (helm and manifests) if it is not removed automatically?

[kate-osborn]

more comments: (1) Would it make sense to add something about leader election to the architecture doc?

(2) What removes the leader election lock resource? Do we need to update our deinstallation instructions (helm and manifests) if it is not removed automatically?

(1) Sure, I can add something to the architecture doc.
(2). It's removed when you delete the namespace.

[ciarams87]

🚀

[pleshakov]

👍

[bjee19]

https://github.com/nginxinc/nginx-kubernetes-gateway/blob/main/design/control-data-plane-separation/design.md

There is a line here that is: "Once we add leader election, the control plane will be able to scale.", would we like to change that now?

[kate-osborn]

https://github.com/nginxinc/nginx-kubernetes-gateway/blob/main/design/control-data-plane-separation/design.md

There is a line here that is: "Once we add leader election, the control plane will be able to scale.", would we like to change that now?

No, that's in an old design document. We don't need to keep those up-to-date.

[bjee19]

🚀🚀🚀