Runtime SDK implementation

[sbueringer]

Implementation issue for #5175

Basic Implementation

Initially we want to add a basic implementation, so we can then work better in parallel on more advanced aspects.

Done:

• Add Extension CRD: (✨ Add API types for Runtime SDK ExtensionConfig #6383 @killianmuldoon)
  • Extension CRD + Runtime SDK feature flag + webhook which blocks CRD create/update if flag not set
  • corresponding code / manifest
• Implement catalog (~ POC branch without openapi.go / without GVHToPath + more unit tests): (✨ Implement Catalog for Runtime SDK #6393 @ykakarap)
  • tracking hooks, their request/response types and their OpenAPI schemas
  • conversion
• Basic Extension Controller: (✨ RuntimeSDK: Add extension config controller skeleton #6411 @killianmuldoon)
  • setup in main.go with corresponding concurrency flag
  • alias files + controller skeleton (e.g. SetupWithManager, no code in Reconcile yet)
• Implement registry: (✨ RuntimeSDK: add registry #6449 @sbueringer)
  • thread-safe storage of registry entries (i.e. name, gvh, connection data, namespaceSelector, failurePolicy)
  • registry should be created only once per binary
• Client: Discovery: (✨ Runtime SDK client and Discovery #6455 @ykakarap)
  • API package with Discovery types
  • DeepCopy + Conversion code generation
  • Discovery implementation in the Client package
• Extension Controller: Discovery ( 🌱 integrate controller for ExtensionConfig #6510 @killianmuldoon)
• Client: Calling one and multiple RuntimeExtensions (incl. aggregation) (✨ Implement Call and CallAll in RuntimeClient #6499 @ykakarap @killianmuldoon )
  • RuntimeSDK: Consider using shared structs in RuntimeHook API types (@killianmuldoon) ✨ Implement Call and CallAll in RuntimeClient #6499
• Implement aggregation (incl. introduce RetryableResponse) (🌱 RuntimeSDK: Implement aggregateResponse for Runtime client #6581 @killianmuldoon @sbueringer)

These are just the first steps, the goal is - as mentioned above - to get the basics in place so we can parallelize better. We have a list of additional aspects we want to address after the basic implementation is done:

Additional Tasks (non-blocking for hook implementations)

Done:

• Advanced defaulting and validation for ExtensionConfig (✨ webhooks: defaulting and validation for ExtensionConfig #6475 @chrischdi)
• Runtime openapi-gen (✨ introduce runtime-openapi-gen to generate openapi spec on release  #6462 @chrischdi)
• Validation of Discovery responses: 🌱 Add client defaulting and validation for DiscoveryResponse #6578 (@killianmuldoon)
• Handle ExtensionConfig.spec.namespaceSelector (🌱 Runtime sdk/namespace selector #6619 @killianmuldoon)
• RuntimeSDK: seamless certificate handling for RuntimeExtensions, includes:
  • Implement a final version of http/https support including webhook validation (🌱 RuntimeSDK: enforce https for extensions #6645 @chrischdi)
  • Add index for secret => ExtensionConfig lookup (🌱 RuntimeSDK: add index for ExtensionConfigs having ca injection annotation set #6648 @chrischdi)
• Check if/which packages we have to export to enable writing a RuntimeExtension more easily (e.g. catalog)
  • Issue: Ease Runtime Extension implementation by providing exported util packages #6720
• Documentation:
  • Propagate RuntimeSDK feature flag in docs (📖 Add basic overview doc and feature flags #6538 @killianmuldoon)
  • Book documentation for RuntimeSDK + hooks (📖 Runtime SDK: extend documentation & update proposals accordingly #6756 @sbueringer)
  • Add disclaimer: clusterctl alpha topology dry run (📖 add disclaimer about clusterctl topology dryrun RuntimeSDK support #6830 @sbueringer)
  • Improve Runtime Hook Swagger documentation (🌱 Improve open API docs #6878 @fabriziopandini )
• General concerns:
  • metrics (🌱 Add metrics to Runtime SDK hook client #6675 @chrischdi)
  • error wrapping/context + godoc comments (🌱 RuntimeSDK: improve error wrapping and godoc #6693 @sbueringer)
  • logging (🌱 runtime-sdk: add logs #6832 @ykakarap)
• Testing:
  • Evaluate test & integration test coverage
  • Some manual testing #6907

Cleanup:

• Consider dedup lowestNonZeroRetryAfterSeconds: Consider dedup lowestNonZeroRetryAfterSeconds #6945
• Open issue to discuss adding constants for field paths (Cluster/MD/...): see ✨ Topology Mutation: use API types for inline patching #6289 (comment): Consider adding constants for field paths of core CAPI resources #6946
• Introduce constants for builtin variable paths: see ✨ Topology Mutation: use API types for inline patching #6289 (comment): Consider introducing constants for builtin variable paths #6947

We have some more additional tasks for later in the Runtime SDK Implementation Design

Follow-up:

• Topology Mutation Hook implementation #6545
• RuntimeHooks for Addon-Management implementation #6546

/kind feature

[sbueringer]

/milestone v1.2

[killianmuldoon]

/area runtime-sdk

[sbueringer]

Created issues for the last open cleanup tasks.

Thx everyone for the great work!!

/close

[k8s-ci-robot]

@sbueringer: Closing this issue.

In response to this:

Created issues for the last open cleanup tasks.

Thx everyone for the great work!!

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.