implement KubernetesExecutor

[adwk67]

Description

Implements KubernetesExecutor.
Fixes #2.
Jenkins: 🟢 https://ci.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/airflow-operator-it-custom/99/

Implementation summary:

• the executor field has been removed from the top-level cluster config
• the workers role has been replaced with a complex enum specifying one of:
  • celery: this contains the worker config/roles etc.
  • kubernetes: this contains just the logging/resources of the config, plus env-overrides (has to be included as we don't have a role object)
• when executor = kubernetes, a pod template is deployed as config map that airflow will use to spawn pods for the DAG tasks

Tests on Openshift:

--- PASS: kuttl (3580.00s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/logging_airflow-2.4.1_openshift-true (282.85s)
        --- PASS: kuttl/harness/ldap_airflow-latest-2.6.1_ldap-authentication-insecure-tls_openshift-true_executor-celery (321.02s)
        --- PASS: kuttl/harness/mount-dags-gitsync_airflow-latest-2.6.1_openshift-true_executor-kubernetes (215.58s)
        --- PASS: kuttl/harness/logging_airflow-2.2.5_openshift-true (305.99s)
        --- PASS: kuttl/harness/logging_airflow-2.2.4_openshift-true (328.94s)
        --- PASS: kuttl/harness/mount-dags-configmap_airflow-latest-2.6.1_openshift-true_executor-kubernetes (202.57s)
        --- PASS: kuttl/harness/mount-dags-configmap_airflow-latest-2.6.1_openshift-true_executor-celery (290.25s)
        --- PASS: kuttl/harness/orphaned-resources_airflow-latest-2.6.1_openshift-true (318.29s)
        --- PASS: kuttl/harness/smoke_airflow-2.2.5_openshift-true_executor-kubernetes (292.73s)
        --- PASS: kuttl/harness/smoke_airflow-2.4.1_openshift-true_executor-kubernetes (267.12s)
        --- PASS: kuttl/harness/smoke_airflow-2.6.1_openshift-true_executor-celery (284.75s)
        --- PASS: kuttl/harness/smoke_airflow-2.4.1_openshift-true_executor-celery (335.22s)
        --- PASS: kuttl/harness/smoke_airflow-2.6.1_openshift-true_executor-kubernetes (226.21s)
        --- PASS: kuttl/harness/cluster-operation_airflow-latest-2.6.1_openshift-true (303.22s)
        --- PASS: kuttl/harness/resources_airflow-latest-2.6.1_openshift-true (209.99s)
        --- PASS: kuttl/harness/smoke_airflow-2.2.4_openshift-true_executor-celery (343.50s)
        --- PASS: kuttl/harness/smoke_airflow-2.2.5_openshift-true_executor-celery (395.81s)
        --- PASS: kuttl/harness/smoke_airflow-2.2.4_openshift-true_executor-kubernetes (314.30s)
        --- PASS: kuttl/harness/logging_airflow-2.6.1_openshift-true (239.35s)
        --- PASS: kuttl/harness/ldap_airflow-latest-2.6.1_ldap-authentication-server-verification-tls_openshift-true_executor-celery (311.55s)
        --- PASS: kuttl/harness/ldap_airflow-latest-2.6.1_ldap-authentication-no-tls_openshift-true_executor-kubernetes (202.59s)
        --- PASS: kuttl/harness/ldap_airflow-latest-2.6.1_ldap-authentication-no-tls_openshift-true_executor-celery (296.19s)
        --- PASS: kuttl/harness/ldap_airflow-latest-2.6.1_ldap-authentication-insecure-tls_openshift-true_executor-kubernetes (246.54s)
        --- PASS: kuttl/harness/mount-dags-gitsync_airflow-latest-2.6.1_openshift-true_executor-celery (251.68s)
        --- PASS: kuttl/harness/ldap_airflow-latest-2.6.1_ldap-authentication-server-verification-tls_openshift-true_executor-kubernetes (200.25s)

Definition of Done Checklist

• Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
• Please make sure all these things are done and tick the boxes

Author

Beta Give feedback

1. Changes are OpenShift compatible
2. Helm chart can be installed and deployed operator works
3. Integration tests passed (add new ones for the executor)
4. Documentation
5. Logging works for ephemeral worker pods (not working: covered by Capture worker logs  #318)
6. Check tini invocation
7. CRD changes are approved
8. Ensure vector container is shutdown on the worker pods used by the kubernetes executor. See wrap entrypoint script to trigger pre/post hook commands docker-images#435

Reviewer

Beta Give feedback

1. Code contains useful comments
2. (Integration-)Test cases added
3. Documentation added or updated
4. Changelog updated
5. Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

Beta Give feedback

1. Feature Tracker has been updated
2. Proper release label has been added

[adwk67]

Jenkins after refactoring CRD structure: 🟢 https://ci.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/airflow-operator-it-custom/100/

[razvan]

It works and it's ok but IMO:

• ExecutorConfig vs AirflowConfig - the names are confusing don't reflect the intent.
• Not sure if it's wise to introduce so much complication just because of the affinity issue of executors. But fortunately it's an implementation detail that we can hopefully change later without breaking the CRD.

[razvan]

This is almost adding support for complete new product. It may be worth adding an explicit integration test for kubernetes workers and actually run a some tasks with it. This is already done.