Issue with x_kubernetes_preserve_unknown_fields in java com.pulumi.kubernetes.apiextensions.v1.CustomResourceDefinition

[elonazoulay]

What happened?

I am attempting to create a CustomResourceDefinition that has a field with x-kubernetes-preserve-unknown-fields set to true via the x_kubernetes_preserve_unknown_fields method and expected the resource to be created but instead get the following error:

Caused by: java.lang.IllegalArgumentException: Expected type 'com.pulumi.kubernetes.apiextensions.v1.outputs.JSONSchemaProps' (annotated with 'com.pulumi.core.annotations.CustomType') to have a setter annotated with @com.pulumi.core.annotations.CustomType$Setter("x-kubernetes-preserve-unknown-fields"), got: $schema,nullable,x_kubernetes_int_or_string,minLength,pattern,description,type,title,required,example,exclusiveMaximum,patternProperties,allOf,not,default,oneOf,additionalItems,x_kubernetes_embedded_resource,id,maxProperties,exclusiveMinimum,x_kubernetes_validations,definitions,multipleOf,maxItems,x_kubernetes_list_type,format,anyOf,enum,minProperties,dependencies,minItems,x_kubernetes_preserve_unknown_fields,x_kubernetes_list_map_keys,uniqueItems,maximum,additionalProperties,x_kubernetes_map_type,externalDocs,minimum,$ref,items,maxLength,properties
    	at com.pulumi.serialization.internal.Converter.lambda$tryConvertObjectInner$7(Converter.java:401)
    	at java.base/java.util.HashMap.forEach(HashMap.java:1429)
    	at com.pulumi.serialization.internal.Converter.tryConvertObjectInner(Converter.java:399)
    	at com.pulumi.serialization.internal.Converter.tryConvertMap(Converter.java:665)
    	at com.pulumi.serialization.internal.Converter.tryConvertObjectInner(Converter.java:281)
    	at com.pulumi.serialization.internal.Converter.lambda$tryConvertObjectInner$7(Converter.java:426)
    	... 34 more

If I remove the fields in the example below that contain x_kubernetes_preserve_unknown_fields it works.

Example

This is the removed portion of the spec:

.put("coordinator", JSONSchemaPropsArgs.builder()
                                                                                        .type("object")
                                                                                        .properties(ImmutableMap.<String, JSONSchemaPropsArgs>builder()
                                                                                                .put("etcConfigFiles", JSONSchemaPropsArgs.builder()
                                                                                                        .x_kubernetes_preserve_unknown_fields(true)
                                                                                                        .build())
                                                                                                .buildOrThrow())
                                                                                        .build())

Which is the equivalent of the following yaml (which does work with kubectl apply -f):

                    etcConfigFiles:
                      x-kubernetes-preserve-unknown-fields: true
                      type: object                   

Output of pulumi about

CLI
Version 3.137.0
Go Version go1.23.2
Go Compiler gc

Plugins
KIND NAME VERSION
language java unknown

Host
OS darwin
Version 14.6.1
Arch arm64

This project is written in java: executable='/Users/elon.azoulay/.jenv/shims/java' version='openjdk 22.0.2 2024-07-16
OpenJDK Runtime Environment Temurin-22.0.2+9 (build 22.0.2+9)
OpenJDK 64-Bit Server VM Temurin-22.0.2+9 (build 22.0.2+9, mixed mode)' maven='Apache Maven 3.9.9 ()' gradle='8.11' java='/Users/elon.azoulay/.jenv/shims/java' javac='22.0.2'

Backend
Name pulumi.com
URL https://app.pulumi.com/elonazoulay
User elonazoulay
Organizations elonazoulay, starburstdata
Token type personal

No dependencies found

Pulumi locates its logs in /var/folders/5q/524n82557j5_bh2vy4pwr0d00000gq/T/ by default

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[blampe]

@elonazoulay what version of pulumi-kubernetes are you using, and can you please include a minimal reproducible example?

[EronWright]

Simply put, the Setter annotation seems to be incorrect (should be @CustomType.Setter("x-kubernetes-preserve-unknown-fields")):

pulumi-kubernetes/sdk/java/src/main/java/com/pulumi/kubernetes/apiextensions/v1/outputs/JSONSchemaProps.java

Lines 716 to 721 in d440f53

	@CustomType.Setter
	public Builder x_kubernetes_preserve_unknown_fields(@Nullable Boolean x_kubernetes_preserve_unknown_fields) {
	this.x_kubernetes_preserve_unknown_fields = x_kubernetes_preserve_unknown_fields;
	return this;
	}

[blampe]

Repro

package myproject;

import com.pulumi.Context;
import com.pulumi.Pulumi;

import com.pulumi.kubernetes.apiextensions.v1.CustomResourceDefinition;
import com.pulumi.kubernetes.apiextensions.v1.CustomResourceDefinitionArgs;
import com.pulumi.kubernetes.apiextensions.v1.inputs.CustomResourceDefinitionNamesArgs;
import com.pulumi.kubernetes.apiextensions.v1.inputs.CustomResourceDefinitionVersionArgs;
import com.pulumi.kubernetes.apiextensions.v1.inputs.CustomResourceDefinitionSpecArgs;
import com.pulumi.kubernetes.apiextensions.v1.inputs.CustomResourceValidationArgs;
import com.pulumi.kubernetes.meta.v1.inputs.ObjectMetaArgs;
import com.pulumi.kubernetes.apiextensions.v1.inputs.JSONSchemaPropsArgs;
import java.util.HashMap;

public class Main {

    public static void main(String[] args) {
        Pulumi.run(Main::stack);
    }

    public static void stack(Context ctx) {
        var metadata = ObjectMetaArgs.builder()
                .name("crds.example.com")
                .build();

        var spec = CustomResourceDefinitionSpecArgs.builder()
                .group("example.com")
                .scope("Namespaced")
                .names(CustomResourceDefinitionNamesArgs.builder()
                        .kind("MyCRD")
                        .plural("mycrds")
                        .singular("mycrd")
                        .shortNames("mycrd")
                        .build())
                .versions(CustomResourceDefinitionVersionArgs.builder()
                        .name("v1")
                        .served(true)
                        .storage(true)
                        .schema(CustomResourceValidationArgs.builder()
                                .openAPIV3Schema(JSONSchemaPropsArgs.builder()
                                        .type("object")
                                        .properties(new HashMap<String, JSONSchemaPropsArgs>() {
                                            {
                                                put("key", JSONSchemaPropsArgs.builder()
                                                        .x_kubernetes_preserve_unknown_fields(true)
                                                        .build());
                                            }
                                        }).build())
                                .build())
                        .build())
                .build();

        new CustomResourceDefinition("crd",
                CustomResourceDefinitionArgs.builder()
                        .metadata(metadata)
                        .spec(spec)
                        .build());
    }
}

[blampe]

Properties containing punctuation (like hyphens) aren't generally supported due to pulumi/pulumi#15874. In other words I don't think we can fix this for Java without also breaking Node and Dotnet.

As a workaround, you could specify the CRD as a YAML manifest and apply it via something like a ConfigFile resource.

[elonazoulay]

Thanks! I tried the config file and config group resources and both had the same error with as this. Using CustomResource works, but it's pretty ugly:)

[blampe]

@elonazoulay can you provide an example of how you were using ConfigFile/ConfigGroup? I'm surprised this gave you the same error since these simply apply YAML manifests. Do these manifests apply cleanly with kubectl?

[EronWright]

Here's a repro, given a prior installation of the PKO CRDs:

package myproject;

import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.kubernetes.apiextensions.v1.CustomResourceDefinition;

public class App {
    public static void main(String[] args) {
        Pulumi.run(ctx -> {
            var crd = CustomResourceDefinition.get("test", Output.of("workspaces.auto.pulumi.com"), null);
            ctx.export("urn", crd.urn());
        });
    }
}

Fails with:

Caused by: java.lang.IllegalArgumentException: Expected type 'com.pulumi.kubernetes.apiextensions.v1.outputs.JSONSchemaProps' (annotated with 'com.pulumi.core.annotations.CustomType') to have a setter annotated with @com.pulumi.core.annotations.CustomType$Setter("x-kubernetes-map-type"), got: $schema,nullable,x_kubernetes_int_or_string,minLength,pattern,description,title,type,required,example,patternProperties,exclusiveMaximum,allOf,oneOf,default,not,x_kubernetes_embedded_resource,additionalItems,maxProperties,id,x_kubernetes_validations,exclusiveMinimum,definitions,multipleOf,maxItems,x_kubernetes_list_type,format,anyOf,enum,minProperties,dependencies,x_kubernetes_preserve_unknown_fields,minItems,x_kubernetes_list_map_keys,uniqueItems,maximum,x_kubernetes_map_type,externalDocs,additionalProperties,minimum,$ref,items,properties,maxLength
        at com.pulumi.serialization.internal.Converter.lambda$tryConvertObjectInner$7(Converter.java:401)
        at java.base/java.util.HashMap.forEach(HashMap.java:1421)
        at com.pulumi.serialization.internal.Converter.tryConvertObjectInner(Converter.java:399)
        at com.pulumi.serialization.internal.Converter.tryConvertMap(Converter.java:665)
        at com.pulumi.serialization.internal.Converter.tryConvertObjectInner(Converter.java:281)
        at com.pulumi.serialization.internal.Converter.lambda$tryConvertObjectInner$7(Converter.java:426)
        ... 39 more

[rquitales]

I’ve identified the root cause of the issue and have a fix in progress. The problem lies in how we handle normalization for x-kubernetes-* fields. During normalization, these fields are converted to x_kubernetes_*. However, after a create/read/update operation, the API Server returns the object with the original x-kubernetes-* format, and this is what gets stored in the state. While this mismatch generally doesn’t cause issues in most languages, it prevents us from accessing the value of crd.spec.versions[0].schema.openAPIV3Schema.x_kubernetes_preserve_unknown_fields as an output because the field name isn’t normalized.

In Java, this causes a runtime failure because our builder doesn’t have a setter for x-kubernetes-*, leading to an error when accessing the output. Notice how the type error arises from the output variant:

Caused by: java.lang.IllegalArgumentException: Expected type 'com.pulumi.kubernetes.apiextensions.v1.outputs...'

Here’s a minimal TypeScript program that reproduces the issue when attempting to access the x_kubernetes_preserve_unknown_fields value:

import * as k8s from "@pulumi/kubernetes";

const crd = new k8s.apiextensions.v1.CustomResourceDefinition("my-cr", {
  metadata: {
    name: "books.example.com",
  },
  spec: {
    group: "example.com",
    names: {
      kind: "Book",
      listKind: "BookList",
      plural: "books",
      singular: "book",
    },
    scope: "Namespaced",
    versions: [
      {
        name: "v1",
        served: true,
        storage: true,
        schema: {
          openAPIV3Schema: {
            type: "object",
            x_kubernetes_preserve_unknown_fields: true,
            properties: {
              apiVersion: { type: "string" },
              kind: { type: "string" },
              metadata: {
                type: "object",
                properties: { name: { type: "string" } },
              },
              spec: {
                type: "object",
                properties: {
                  title: { type: "string" },
                },
              },
            },
          },
        },
      },
    ],
  },
});

export const outputCRD = crd;

// This output will be masked.
export const nestedPreserve = crd.spec.versions[0].schema.openAPIV3Schema.x_kubernetes_preserve_unknown_fields;

The following error occurs:

warning: Undefined value (nestedPreserve) will not show as a stack output.

[rquitales]

Note: the ideal fix would be to have custom setters to solve this, however, we don't have enough codegen plumbing to support this for all languages so we'll need to continue with the (de)normalization route to address this that is not Java specific.

[rquitales]

@elonazoulay We just released v4.18.4 of the Pulumi Kubernetes provider which includes this fix. Please let us know if you encounter any other issues. Thanks!

[elonazoulay]

@elonazoulay We just released v4.18.4 of the Pulumi Kubernetes provider which includes this fix. Please let us know if you encounter any other issues. Thanks!

Amazing! Thanks so much @rquitales!