Support NGINX configuraton for production use

[pleshakov]

Summary

Problem statement

How can we support NGINX configuration not available in the Gateway API, but relevant for production use? So that our users can start using NGINX Kubernetes Gateway (NKG) in their production environments?

This discussion is to:

1. Acknowledge the problem.
2. Narrow down its scope.
3. Find an appropriate solution.

Out of Scope

We will focus on NGINX configuration related to basic proxying features available through Gateway API HTTPRoute and Gateway resources rather than advanced features like authentication and rate limiting.

Background

Currently, NKG supports features reflected in https://github.com/nginxinc/nginx-kubernetes-gateway/blob/1df6bcd08129c687178a38f880f95871fb5275c2/docs/gateway-api-compatibility.md

Supporting those features does not mean NKG is suitable for production use, as many NGINX configuration settings related to those features are not available to configure through the Gateway API resources.

Below is a list of such settings:

• Backend related settings
  • Timeouts
  • Retries
  • Load balancing method
  • Passive health checks
  • Active health checks
  • Keepalives
  • Buffering
  • Body size
• Client related settings
  • TLS termination parameters
  • HTTP/2
  • Timeouts
  • Keepalives
  • Proxy protocol
  • Extracting client IP from fronting proxy
• Logging
  • Log format
  • Log level
• NGINX performance
  • number of worker processes and CPU affinity
  • worker shutdown timeout
  • connections per worker

Note: a judgment call on importance of those settings can be made by looking into the early changelog of the NGINX Ingress Controller and seeing when those were added. The features were brought largely by user requests and feedback.

Additionally, sometimes, it is necessary to tune some aspect of NGINX behavior from a broader number of NGINX directives. For example, configure the backlog parameter of the listen directive for high performance optimization. In NKG, we already support the listen directive - it is necessary for NGINX to accept client traffic -- but we don't expose the listen directive parameters.

What do we already have in NKG?

So far, NKG only supports features configurable through Gateway API resources. For many of those features, NKG either chooses a default or uses the NGINX default for the settings mentioned above.

Additionally, NKG creates the main NGINX configuration file from this ConfigMap -- https://github.com/nginxinc/nginx-kubernetes-gateway/blob/1df6bcd08129c687178a38f880f95871fb5275c2/deploy/manifests/nginx-conf.yaml#L8-L20 -- so a number of those mentioned above settings can be supported by editing that ConfigMap.

Can Gateway API support those settings?

Evolution of Gateway API

Some of those settings, although in limited form (because of data plane differences) will (or might) be supported in the Gateway API:

• Timeouts -- GEP: Timeouts kubernetes-sigs/gateway-api#1742 and GEP-1742: HTTPRoute Timeouts API kubernetes-sigs/gateway-api#1997
• Retries -- Configurable Retries in HTTPRoute kubernetes-sigs/gateway-api#1731
• Load balancing stategy -- Add support for Load Balancing Policy kubernetes-sigs/gateway-api#1778

Extensibility of Gateway API

The Gateway API comes with a number of extension points that NKG can take advantage of to expose additional NGINX-specific settings. See https://gateway-api.sigs.k8s.io/concepts/api-overview/#extension-points

Additionally, the GatewayClass resource can reference a ConfigMap or an implementation specific resource that can define implementation-specific configuration -- https://gateway-api.sigs.k8s.io/api-types/gatewayclass/#gatewayclass-parameters

Gateway resource will reference an implementation specific configuration once this GEP https://gateway-api.sigs.k8s.io/geps/gep-1867/ is implemented.

Existing Solutions in Related Projects

In NGINX Ingress Controller (NIC) those settings are part of its configuration API:

• See annotations -- https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/
• See ConfigMap -- https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/
• See VirtualServer resource, its upstream settings -- https://docs.nginx.com/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/#upstream

Additionally, NIC comes with advanced configuration method to support settings not available through the configuration API:

• Snippets -- https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-snippets/
• Custom templates -- https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/custom-templates/
• Custom annotations -- https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/custom-annotations/

[brianehlert]

I would first probably make the subject more descripting of the problem being solved. As "configuration for production use" is pretty vague.

Is the intention to support some type of 'snippets' functionality?
By snippets I mean - the user has some advanced directive they want to use in a specific nginx.conf context (http/stream, server, location, upstream) and that / those directives are not already expressed in the YAML objects.

My assumption would be both that the user has knowledge of nginx.conf as well as they would not try to bypass the API objects entirely (and thus the value of a controller) by simply presenting a complete nginx.conf.

[pleshakov]

I would first probably make the subject more descripting of the problem being solved. As "configuration for production use" is pretty vague.

@brianehlert I gave the list of features in the Background section

Is the intention to support some type of 'snippets' functionality?

snippets is a possible solution. I haven't suggested any solution yets

My assumption would be both that the user has knowledge of nginx.conf as well as they would not try to bypass the API objects entirely (and thus the value of a controller) by simply presenting a complete nginx.conf.

👍

[brianehlert]

My comment on the title was intended to drive eyeballs through being more descriptive about the problem.
The subject itself does not seem to indicate a problem to solve, it reads as if this is a necessary improvement as if the current configuration output of the controller is not "production use".
Humans will scan the subject lines to decide whether to click in.

Trying to solve the problem of "allowing the defining NGINX functionality that the Gateway API does not yet support" is a more tangible and specific problem.

[pleshakov]

it reads as if this is a necessary improvement as if the current configuration output of the controller is not "production use".

production use is the reason to support those configuration settings. those settings are relevant for production use.

Trying to solve the problem of "allowing the defining NGINX functionality that the Gateway API does not yet support" is a more tangible and specific problem

This problem is much broader though. NGINX has so many features and settings. That's why I added an out of scope statement "We will focus on NGINX configuration related to basic proxying features available through Gateway API HTTPRoute and Gateway resources rather than advanced features like authentication and rate limiting." - so that the discussion stays focused on supporting existing Gateway API core features in the production ready way.

[brianehlert]

I honestly don't think we are saying things that are very different.

For me; The term "production use" is a judgement statement of why a customer would need to apply a necessary setting in a context and that setting directly defined as a parameter in the object model.

All we know is that there is an NGINX directive that customers will want to use in an nginx.conf context and we can't attempt to light up every obscure directive through YAML and the corresponding logic.

[brianehlert]

The NIC capabilities of snippets, custom templates, and custom annotations was mentioned.
There is another way that NIC is customized, and that the modifying the templates directly. There are customers that do this. We can't ignore that it happens.

While custom templates, custom annotations, and template customization are possible - they tend to limit and restrict the ability for users to upgrade. Upgrades frequently result in broken configurations. Or the upgrade cycle is long for these customers to validate and update for each release which results in the customers not keeping up to date.
As a result we have tended to not encourage custom templates and annotations.

The snippets style is a pattern has survived well through upgrades. Combine this with reliable NGINX behavior from directives and the customers have had a relatively painless experience with this pattern.

[sjberman]

For the functionality that can be updated by a user directly in the initialization ConfigMap, we can document a user's ability to do this. We would need to be clear that this is not dynamic and will only be set at deploy-time.

For backend-specific settings that cannot be set at deploy time, we can rely on nginx defaults, or update the defaults to values we think are better. This would cover most general cases for users until we allow more fine-tuning for the users that will get into the weeds more.

If we truly want to expose a general way to inject nginx config at v1.0, like snippets, we need to consider security implications and treat them as a feature, and not just a "hack" for users to update configuration until we have an official API for them, because users will take this feature and run with it. Support and engineering will be on the hook to help diagnose issues with it. This all means that it should be robust from the start so we don't break things to make it better in a future release. Which leads to the question of: do we have time for that compared to our current list of must-haves for v1.0?

[pleshakov]

For backend-specific settings that cannot be set at deploy time, we can rely on nginx defaults, or update the defaults to values we think are better. This would cover most general cases for users until we allow more fine-tuning for the users that will get into the weeds more.

good thing that timeouts, retries, buffering, body size can also be supported in the configmap, because you can set the defaults in the http context

[blaisewang]

As a security company that provides third-party Nginx modules, we currently rely heavily on snippet capabilities in our Nginx Ingress solutions. We hope that the Nginx Gateway Fabric can offer unrestricted snippet capabilities, rather than only allowing certain specific commands to be used. I believe that disabling snippet capabilities by default is a good choice, and they should only be enabled once the user is fully aware of the risks/consequences.

[mpstefan]

Hey @blaisewang, thanks for the feedback!

We actually just discussed this functionality today, and we're planning out some design work for snippet-like functionality in our next release, likely implemented in the release after that. Keep an eye out for our enhancement proposal for this in around 2 months time.

[pleshakov]

Closing this because #1566 and https://github.com/nginxinc/nginx-gateway-fabric/blob/main/docs/proposals/nginx-extensions.md will address the mentioned production use cases.

[Okm165]

@pleshakov Do u know when body size setting will be available on production?
It is blocking me rn on my kubernetes service
Cheers ;)

[pleshakov]

@Okm165
it is planned for 1.3.0 , which is planned for end of May
#1245
https://github.com/nginxinc/nginx-gateway-fabric/milestone/17

It will be available through a policy - https://github.com/nginxinc/nginx-gateway-fabric/blob/main/docs/proposals/client-settings.md

there is a workaround to use it now mentioned in #1245 (comment)