kube-derive does not create required schemas field for CustomResourceDefinition

[praveenperera]

With apiextensions.k8s.io/v1 the definition of a structural schema is mandatory for CustomResourceDefinitions, while in v1beta1 this is still optional.

ref: https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/

Therefore should kube-derive only be allowed to create apiextensions.k8s.io/v1beta1 CRD files?

[clux]

Oh, interesting. I seem to recall testing and managed to get it working with the stable v1 on 1.18 and 1.17 without a schema, but will do some more testing.

At any rate, it is possible that sticking to v1beta1 is a reasonable stop-gap solution until we have schemas (#129)

[praveenperera]

Note: Kubernetes 1.19 will stop serving apiextensions.k8s.io/v1beta1, and is set to be released August 4th 2020

Nevermind removal has been moved to 1.22, so we got time

ref: kubernetes/kubernetes#82022 (comment)

[sdlarsen]

Hmm. With kube 0.40 and 1.18 I get this when using #[kube(apiextensions = "v1beta1")] on my CRD:

apply error: ErrorResponse { status: "Failure", message: "Incorrect version specified in apply patch. Specified patch version: apiextensions.k8s.io/v1beta1, expected: apiextensions.k8s.io/v1", reason: "BadRequest", code: 400 }

is anything else needed for this?

[clux]

Hm. It shouldn't need anything else. It could be a bug. You could try to grab your crd via Foo::crd() and drop it here.

Oh, and if you are posting it to kubernetes via a Api<CustomResourceDefinition> make sure you are using the old CRD type:

use k8s_openapi::apiextensions_apiserver::pkg::apis::apiextensions::v1beta1::CustomResourceDefinition;

not the stable one ^

[sdlarsen]

@clux, thanks, I was using the stable CRD type...

[clux]

Ah, good. Yeah, that is a bit of a footgun at the moment...

[Pscheidl]

Unfortunately, this problem is still present in 0.43.0 version.

I have a custom resource:

#[derive(CustomResource, Debug, Clone, Deserialize, Serialize)]
#[kube(group = "h2o.ai", version = "v1", kind = "H2O", namespaced)]
#[kube(shortname = "h2o", namespaced)]
pub struct H2OSpec {
    pub nodes: u32,
    pub resources: Resources,
}

#[derive(Debug, Clone, Deserialize, Serialize)]
pub struct Resources {
    pub cpu: u32,
    pub memory: String,
    #[serde(rename = "memoryPercentage", skip_serializing_if = "Option::is_none")]
    pub memory_percentage: Option<u8>,
}

And the output of

    let h2o_crd: CustomResourceDefinition = H2O::crd();
    println!("{}", serde_yaml::to_string(&h2o_crd).unwrap());

is

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: h2os.h2o.ai
spec:
  group: h2o.ai
  names:
    kind: H2O
    plural: h2os
    shortNames:
      - h2o
    singular: h2o
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true

There is no schema, therefore when trying to create the resource in Kuernetes, a validation error occurs:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Api(ErrorResponse { status: "Failure", message: "CustomResourceDefinition.apiextensions.k8s.io \"h2os.h2o.ai\" is invalid: spec.versions[0].schema.openAPIV3Schema: Required value: schemas are required", reason: "Invalid", code: 422 })'

Fortunately, the workaround is easy, just create the YAML myself:

const H2O_RESOURCE_TEMPLATE: &str = r#"
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: h2os.h2o.ai
spec:
  group: h2o.ai
  names:
    kind: H2O
    plural: h2os
    singular: h2o
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                nodes:
                  type: integer
                resources:
                  type: object
                  properties:
                    cpu:
                      type: integer
                      minimum: 1
                    memory:
                      type: string
                      pattern: "^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$"
                    memoryPercentage:
                      type: integer
                      minimum: 1
                      maximum: 100
                  required: ["cpu", "memory"]
              required: ["nodes", "resources"]
"#;

And the create the CRD manually with a simple call to serde:

return serde_yaml::from_str(H2O_RESOURCE_TEMPLATE).unwrap();

[kazk]

Is anyone working on this? I thought schemars could be used:

// in crd()
let schema = schemars::schema_for!(Self);
// definitions is a map of struct's name to schema.
let spec_schema = &schema.definitions.get(stringify!(#ident)).unwrap();
serde_json::json!({
    // ...
    "spec": {
        // ...
        "versions": [{
            // ...
            "schema": {
                "openAPIV3Schema": {
                    "type": "object",
                    "properties": {
                        "spec": spec_schema,
                    }
                }
            }
        }],
    }
})

(Requires adding ::schemars::JsonSchema to derive_paths and #[schemars(skip)] to metadata field.)

However, this fails when there are optional fields (e.g., info: Option<String>) because schemars outputs type: ["string", null] which is valid in JSON Schema (what schemars is for), but not in OpenAPI v3 Schema (see JSONSchemaProps's type_).
Maybe there's a crate similar to schemars for Open API v3?

I also found k8s_openapi_derive::CustomResourceDefinition. They don't derive schema either.

---

@clux Thanks for creating kube. I've been learning Kubernetes controllers, but was overwhelmed by the amount of setup required to write one in Go (code gen, frameworks, etc). Even a minimal one requires a lot of work to get started. kube made it extremely straightforward.
I ported programming-kubernetes/cnat to Rust (cnat-rs) and did some comparisons against their 3 versions (client-go, kubebuilder, operator sdk). I learned a lot from the examples and h2o-kubernetes (thanks @Pscheidl), maybe cnat will be useful to someone.

[MikailBag]

@kazk did you generating schema with custom settings? I think this function should work.

[kazk]

@MikailBag Thanks! Changing to

let gen = schemars::gen::SchemaSettings::openapi3().into_generator();
let schema = gen.into_root_schema_for::<Self>();
let spec_schema = &schema.definitions.get(stringify!(#ident)).unwrap();

worked 🎉

crd_derive example changed apiextensions to v1 and added derive JsonSchema:

use k8s_openapi::Resource;
use kube::CustomResource;
use schemars::JsonSchema;
use serde::{Deserialize, Serialize};

/// Our spec for Foo
///
/// A struct with our chosen Kind will be created for us, using the following kube attrs
#[derive(CustomResource, Serialize, Deserialize, Default, Debug, PartialEq, Clone, JsonSchema)]
#[kube(
    group = "clux.dev",
    version = "v1",
    kind = "Foo",
    namespaced,
    status = "FooStatus",
    derive = "PartialEq",
    derive = "Default",
    shortname = "f",
    scale = r#"{"specReplicasPath":".spec.replicas", "statusReplicasPath":".status.replicas"}"#,
    printcolumn = r#"{"name":"Spec", "type":"string", "description":"name of foo", "jsonPath":".spec.name"}"#
)]
#[kube(apiextensions = "v1")]
pub struct MyFoo {
    name: String,
    #[serde(skip_serializing_if = "Option::is_none")]
    info: Option<String>,
}

fn main() {
    let crd = serde_json::to_string_pretty(&Foo::crd()).unwrap();
    println!("Foo CRD: \n{}", crd);
}

Output

{
  "apiVersion": "apiextensions.k8s.io/v1",
  "kind": "CustomResourceDefinition",
  "metadata": {
    "name": "foos.clux.dev"
  },
  "spec": {
    "group": "clux.dev",
    "names": {
      "kind": "Foo",
      "plural": "foos",
      "shortNames": [
        "f"
      ],
      "singular": "foo"
    },
    "scope": "Namespaced",
    "versions": [
      {
        "name": "v1",
        "schema": {
          "openAPIV3Schema": {
            "properties": {
              "spec": {
                "description": "Our spec for Foo\n\nA struct with our chosen Kind will be created for us, using the following kube attrs",
                "properties": {
                  "info": {
                    "nullable": true,
                    "type": "string"
                  },
                  "name": {
                    "type": "string"
                  }
                },
                "required": [
                  "name"
                ],
                "type": "object"
              }
            },
            "type": "object"
          }
        },
        "served": true,
        "storage": true
      }
    ]
  }
}