Add Spark Operator into Theia Helm chart #31

yanjunz97 · 2022-06-02T01:26:56Z

This PR adds Spark Operator into Theia Helm chart. Users can install Spark Operator to run our Network Policy Recommendation jobs by enable the spark operator in Theia Helm chart.

Signed-off-by: Yanjun Zhou zhouya@vmware.com

dreamtalen

Thanks Yanjun working on this, helm support will simplify the deployment and usage steps for users of policy recommendation feature. So I think this PR could merge first before the implementation PR.

One thing I'm not sure here is whether let Spark Operator be installed by default.

dreamtalen · 2022-06-02T22:51:34Z

build/charts/theia/values.yaml

@@ -99,3 +99,13 @@ grafana:
    - pod_to_external_dashboard.json
    - node_to_node_dashboard.json
    - networkpolicy_dashboard.json
+sparkOperator:
+  # -- Install the Spark Operator. It is required to run Network Policy Recommendation jobs.
+  enable: false


Could we enable the spark operator by default?

We can do that. If so, do we also update our flow-visibility.yaml to include the Spark Operator? With this, the YAML file users need to apply spark-operator-crds.yaml together with the clickhouse operator before they apply flow-visibility.yaml.

Would also like to hear from @salv-orlando about whether to include it by default.

If so, do we also update our flow-visibility.yaml to include the Spark Operator? With this, the YAML file users need to apply spark-operator-crds.yaml together with the clickhouse operator before they apply flow-visibility.yaml.

I prefer not to include the Spark Operator in flow-visibility.yaml, because we didn't include the clickhouse operator crd either.

The Spark Operator case is a little bit different comparing the ClickHouse Operator, as besides CRDs, it contains some Deployment and Service Account.

If we do not include the Spark Operator in flow-visibility.yaml, there will be a gap that by default, Helm install includes the Spark Operator, but flow-visibility.yaml does not. If the YAML file users would like to enable the Spark Operator, they need apply spark-operator-crds.yaml and generate a flow-visibility.yaml with Spark Operator Deployment by running ./hack/generate_manifest.sh --spark-operator.

I'm not sure whether this may make users confused.

Got it, if so let's keep consistent with current steps. If we are moving forward to include spark operator in helm by default, let's include the spark operator deployments in flow-visibility.yaml and remind yaml file users apply the crd of both clickhouse and spark operator before applying flow-visibility.yaml

my take is that flow-visibility.yaml should be for users that install Antrea the "hard way". These would likely more be users that try to get a good understand of how the various pieces work and integrate each other.
So I do not see as a must having the spark operator in flow-visibility.yaml. But being able to add it won't do any harm.

Every other user would like to install Antrea (and manage it after installation) in the simplest possible way, and they will likely go for helm charts.

Got it, also according to our discussion in #36, it's ok to let users set mandatory parameters in values.yaml before running helm install. We could keep sparkOperator enable: false here by default.

yanjunz97 · 2022-06-03T01:22:40Z

This PR will introduce some conflicts with PR #25 and PR #27 . The later merged one should take care of the conflicts.

We need to update the Network Policy Recommendation doc to use Theia Helm Chart or our YAML manifests to install the Spark Operator instead of the Official one.
We need to update the label when getting the Spark Operator pod in Theia CTL. The official Spark Operator will have the following labels:
```
app.kubernetes.io/name: spark-operator
app.kubernetes.io/instance: policy-reco
```
These are generate according to some Helm conventions. As instance should be the release name, which is thiea in our case, I removed it for potential conflicts. Would also like to have some suggestions on defining the label.
~~Instead of using a single YAML, e2e test will need to apply spark-operator-crds.yaml and a flow-visibility.yaml with sparkOperator enabled. More details about this flow-visibility.yaml:~~
- ~~If we decide to enable the Spark Operator by default, then we can directly use the flow-visibility.yaml under build/yamls.~~
- ~~If we decide not to enable the Spark Operator by default, this flow-visibility.yaml requires to be generated by ./hack/generate-manifest.sh --spark-operator.~~
e2e test can apply the flow-visibility.yaml with Spark Operator generated by ./generate-manifest.sh --spark-operator

heanlan

overall LGTM

heanlan · 2022-06-03T22:18:50Z

hack/generate-manifest.sh

@@ -24,6 +24,7 @@ _usage="Usage: $0 [--mode (dev|release)] [--keep] [--help|-h]
 Generate a YAML manifest for the Clickhouse-Grafana Flow-visibility Solution, using Helm and
 Kustomize, and print it to stdout.
        --mode (dev|release)  Choose the configuration variant that you need (default is 'dev')
+        --spark-operator      Generate a manifest with spark-operator support enabled


I feel it would be better to highlight that only the Deployment and Service Account will be included in the generated yaml. The crd part will not be included/need to separately apply spark-operator-crds.yaml etc.

Thanks. I added some descriptions, but still feel it might be strange as we do not mention that the ClickHouse CRDs are required before applying flow-visibility in this script.

Actually, I'm considering about injecting the spark-operator-crds.yaml into flow-visibility.yaml with Kustomize in this script when using --spark-operator option. As the installation order does not matter, this might make the deployment easier. Would like to hear some suggestions.

I'm considering about injecting the spark-operator-crds.yaml into flow-visibility.yaml with Kustomize in this script when using --spark-operator option.

I think it's a good idea

I'm considering about injecting the spark-operator-crds.yaml into flow-visibility.yaml with Kustomize in this script when using --spark-operator option.

Sounds good to me.

but still feel it might be strange as we do not mention that the ClickHouse CRDs are required before applying flow-visibility in this script.

Similar cases with the above one, since we already included the ClickHouse deployment in flow-visibility.yaml, what about include ClickHouse CRDs inside flow-visibility.yaml?

We cannot include ClickHouse CRDs inside flow-visibility.yaml, as ClickHouse CRDs must be applied before we use the Kind ClickHouseInstallation.

salv-orlando

I did not follow the story of this PR too much, so I'm not sure if there is still any work in progress. For what is worth, I think it's good enough to merge

yanjunz97 · 2022-06-04T20:09:54Z

I did not follow the story of this PR too much, so I'm not sure if there is still any work in progress. For what is worth, I think it's good enough to merge

This PR is ready for review. It can be merged if no more comment,

Signed-off-by: Yanjun Zhou <zhouya@vmware.com>

yanjunz97 mentioned this pull request Jun 2, 2022

Implementation of theia and policy recommendation CLI #25

Merged

yanjunz97 force-pushed the helm-spark-operator branch 3 times, most recently from 1d62394 to 6b9f279 Compare June 2, 2022 22:35

dreamtalen approved these changes Jun 2, 2022

View reviewed changes

yanjunz97 force-pushed the helm-spark-operator branch from 6b9f279 to 7a110bc Compare June 2, 2022 23:53

dreamtalen mentioned this pull request Jun 3, 2022

Add docs for theia and policy recommendation CLI #36

Merged

yanjunz97 force-pushed the helm-spark-operator branch from 7a110bc to 242748a Compare June 3, 2022 19:03

yanjunz97 requested review from heanlan and salv-orlando June 3, 2022 21:13

dreamtalen added this to the 0.1 milestone Jun 3, 2022

heanlan reviewed Jun 3, 2022

View reviewed changes

salv-orlando reviewed Jun 4, 2022

View reviewed changes

salv-orlando mentioned this pull request Jun 6, 2022

Add e2e test for policy recommendation #27

Merged

yanjunz97 force-pushed the helm-spark-operator branch from 1c4d8ed to eb1a7d9 Compare June 6, 2022 17:34

Add Spark Operator into theia Helm chart

317ef97

Signed-off-by: Yanjun Zhou <zhouya@vmware.com>

yanjunz97 force-pushed the helm-spark-operator branch from eb1a7d9 to 317ef97 Compare June 6, 2022 20:16

yanjunz97 merged commit 275d6be into antrea-io:main Jun 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Spark Operator into Theia Helm chart #31

Add Spark Operator into Theia Helm chart #31

yanjunz97 commented Jun 2, 2022

dreamtalen left a comment

dreamtalen Jun 2, 2022

yanjunz97 Jun 3, 2022

dreamtalen Jun 3, 2022

yanjunz97 Jun 3, 2022

dreamtalen Jun 3, 2022

salv-orlando Jun 4, 2022

dreamtalen Jun 6, 2022

yanjunz97 commented Jun 3, 2022 •

edited

Loading

heanlan left a comment

heanlan Jun 3, 2022

yanjunz97 Jun 3, 2022

heanlan Jun 3, 2022

dreamtalen Jun 4, 2022

yanjunz97 Jun 4, 2022

salv-orlando left a comment

yanjunz97 commented Jun 4, 2022

Add Spark Operator into Theia Helm chart #31

Add Spark Operator into Theia Helm chart #31

Conversation

yanjunz97 commented Jun 2, 2022

dreamtalen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yanjunz97 commented Jun 3, 2022 • edited Loading

heanlan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

salv-orlando left a comment

Choose a reason for hiding this comment

yanjunz97 commented Jun 4, 2022

yanjunz97 commented Jun 3, 2022 •

edited

Loading