Load OpenAPI files from ConfigMap

[anderius]

Is your feature request related to a problem? Please describe.

When reloading the configuration, or starting a Nginx IC pod, all external references must be present and responsive. This puts undesired dependencies on external systems.

There is a warning about that here:

Note:
You need to make sure that the server where the resource files are located is always available when you are compiling your policy.

Describe the solution you'd like

It would be a better option to put the OpenAPI file in a ConfigMap (or a custom resource), and load it from there. Or (if it is not too big), include it directly in the APPolicy (ie base64 encoded).

Additional context

Right now OpenAPI spec is configured like this (source):

apiVersion: appprotect.f5.com/v1beta1
kind: APPolicy
spec:
  policy:
    open-api-files:
      - link: http://${NAME}.aps-ap1841.svc.cluster.local/v3/api-docs

[github-actions]

Hi @anderius thanks for reporting!

Be sure to check out the docs and the Contributing Guidelines while you wait for a human to take a look at this 🙂

Cheers!

[brianehlert]

I think there might be a couple different issues here...

All configuration artifacts being present or able to be processed has always been an NGINX requirement.
This tends to be an issue when DNS resolution of an upstream group is necessary, such as when an externalName service is configured as the backend.
Thus at load/reload (such as startup or config change) the DNS name must be able to be resolved by NGINX.

The page you link mentions the NGINX App Protect WAF Open API spec and I am guessing using an external reference to provide it to NIC + WAF as a proposal to avoid a timing / workflow race condition when using external references.

The external references pattern that is linked to, comes from NGINX App Protect WAF and not the NIC project.
The new method of NGINX App Protect WAF is to use Policy Bundles. Which includes all of the artifacts in a GZIP TAR file that is presented to the ingress controller.

We are working on building additional documentation that better describes this workflow.
It was introduced in WAF 4.x and is the core method of presenting configuration in WAF 5.x
https://docs.nginx.com/nginx-ingress-controller/installation/integrations/app-protect-waf/configuration/#waf-bundles

[anderius]

(Please direct me to another place if that is more suitable for this feature request)

WAF bundles are nice, but to my understanding they only work in one of these conditions:

• The bundles are created up front. That is not possible when every single app has its own policy (openapi spec)
• There is some k8s operator that handles them. That is, reads the OpenAPI spec from the external source, and creates and places the bundle at the correct destination

The need to create a custom operator, only for this, would be less than ideal. :-)

My goal is to support a multi-tenancy scenario, in which the error of one tenant should not stop reloads (or even startup) for all tenants using the same ingress. Right now I consider loading OpenAPI files to be a big threat to that goal.

Other file references has the option to specify file content directly. optionally base64 encoded. For instance the json-validation-files:

        "json-validation-files": [
            {
                "fileName": "person_schema.json",
                "contents": "{\r\n \"$schema\": \"http://json-schema.org/draft-07/schema#\",\r\n \"title\": \"Person\",\r\n \"type\": \"object\",\r\n \"properties\": {\r\n \"firstName\": {\r\n \"type\": \"string\",\r\n \"description\": \"The person's first name.\"\r\n },\r\n \"lastName\": {\r\n \"type\": \"string\",\r\n \"description\": \"The person's last name.\"\r\n },\r\n \"age\": {\r\n \"description\": \"Age in years which must be equal to or greater than zero.\",\r\n \"type\": \"integer\",\r\n \"minimum\": 0\r\n }\r\n }\r\n}"
            }
        ],

[brianehlert]

I think there is a conflation of products here.
You continue to refer to NGINX App Protect WAF. And that is supported in conjunction with (as an add-on to) NGINX Ingress Controller. But the two are not the same.

My goal is to support a multi-tenancy scenario, in which the error of one tenant should not stop reloads (or even startup) for all tenants using the same ingress.

Part of this is separate from the OpenAPI spec and relates to how the system is configured or deployed.
Some customers handle this problem with multiple deployments of NGINX Ingress Controller + NGINX App Protect WAF, thus reducing the potential blast zone of configuration change.
Others handle this by using the VirtualServer / VirtualServerRoute resources instead of the Ingress resource. Since the impact can be contained differently.

Policy Bundles were an NGINX App Protect WAF response to the external reference within App Protect WAF itself.
As the Bundles is local to some volume presented to the ingress controller.

Today, we generally expect customers to be using JSON based Policy with the ingress controller. As that encapsulates the Policy as an object. It has the negative impact on the system in that a Policy compilation happens within the ingress controller container at reload and a corresponding CPU spike which can be prolonged if CPU limits are set.

[ald8]

@anderius

Perhaps file references are exactly what you need?

"open-api-files": [
        {
            "link": "file:///myapi2.json"
        }
    ]

[anderius]

Thank you all for the input!

Using a file reference is of course an option, but that would, as far as I know, require writing tooling for that. I work in an organization with multiple, dedicated platform teams, so that is doable. But, as always, writing nothing at all is usually better. :-)

To be more precise, would it be possible with something like this openapi files as well?

kubernetes-ingress/config/crd/bases/appprotect.f5.com_appolicies.yaml

Lines 774 to 779 in c6f0637

	contents:
	type: string
	fileName:
	type: string
	isBase64:
	type: boolean

Example usage that would be the same for openapi files:

kubernetes-ingress/tests/data/ap-waf-grpc/grpc-block-sayhello.yaml

Lines 26 to 29 in c6f0637

	idl-files:
	- isBase64: true
	fileName: autheid.proto
	contents: Ly8gVGhlIGdyZWV0aW5nIHNlcnZpY2UgZGVmaW5pdGlvbi4KCnN5bnRheCA9ICJwcm90bzMiOwoKcGFja2FnZSBoZWxsb3dvcmxkOwoKc2VydmljZSBHcmVldGVyIHsKICAvLyBTZW5kcyBhIGdyZWV0aW5nCiAgcnBjIFNheUdvb2RieWUgKEhlbGxvUmVxdWVzdCkgcmV0dXJucyAoSGVsbG9SZXBseSkge30KfQoKLy8gVGhlIHJlcXVlc3QgbWVzc2FnZSBjb250YWluaW5nIHRoZSB1c2VyJ3MgbmFtZS4KbWVzc2FnZSBIZWxsb1JlcXVlc3QgewogIHN0cmluZyBuYW1lID0gMTsKfQoKLy8gVGhlIHJlc3BvbnNlIG1lc3NhZ2UgY29udGFpbmluZyB0aGUgZ3JlZXRpbmdzCm1lc3NhZ2UgSGVsbG9SZXBseSB7CiAgc3RyaW5nIG1lc3NhZ2UgPSAxOwp9Cg==

If that functionality is not provided by the ingress controller, please direct me to the correct project.

@brianehlert We already use VirtualServer resources, but I am not aware of how that affects the impact of configuration error. Any doc links for that would be very much appreciated!

[brianehlert]

I was just coming back to this and the file reference is essentially what the origional request is.
The still requires the spec be present.
The spec can be mounted at a known path either as a volume full of specs or as a path per configmap.

It solves the remote reference problem.
There is still timing required that the spec be present before it can be referenced in a Policy.