K8SSAND-1855 ⁃ Cluster level tasks

[jsanda]

What is missing?
cass-operator provides the CassandraTask CRD for performing various operations like rolling restart, cleanup, and rebuild. Here is an example manifest:

apiVersion: control.k8ssandra.io/v1alpha1
kind: CassandraTask
metadata:
  name: upgradesstables
spec:
  datacenter:
    name: dc1
    namespace: k8ssandra-operator
  jobs:
    - name: upgradesstables-dc1
      command: upgradesstables

Creating this CassandraTask will result in cass-operator running upgradesstables against all nodes in dc1.

There needs to be an analogous cluster-level task CRD to run tasks against all DCs. It might look something like this:

apiVersion: control.k8ssandra.io/v1alpha1
kind: K8ssandraClusterTask
metadata:
  name: upgradesstables
spec:
  datacenters:
    - name: dc1
      namespace: k8ssandra-operator
      k8sContext: east
    - name: dc2 
      namespace: k8ssandra-operator
      k8sContext: west
  jobs:
    - name: upgradesstables-dc1
      command: upgradesstables

The datacenters list specifies the DCs on which the task should be run as well as the order in which it should be run. The k8sContext field tells the operator in which Kubernetes cluster the DC exists.

Why do we need it?
Currently if I want to run a task, such as upgradesstables, against a multi-DC cluster I have to some amount of orchestration. k8ssandra-operator should handle this since it already provides cluster-level orchestration and management.

Environment

• K8ssandra Operator version:

  v1.3.0

  **Anything else we need to know?**:

┆Issue is synchronized with this Jira Task by Unito
┆friendlyId: K8SSAND-1855
┆priority: Medium

[adutra]

FYI this feature has become a prerequisite for a cluster-level rebalance task and should be prioritized. cc @adejanovski

[adutra]

@jsanda why do you suggest a list of datacenters instead of a reference to the K8ssandraCluster object itself?

e.g.

apiVersion: control.k8ssandra.io/v1alpha1
kind: K8ssandraClusterTask
metadata:
  name: upgradesstables
spec:
  cluster:
    name: cluster1
    namespace: k8ssandra-operator
  jobs:
    - name: upgradesstables-dc1
      command: upgradesstables

My fear is that users would end up listing DCs that do not belong to the same clusters, in which case a task like rebalance would produce undetermined results.

[jsanda]

I suggested a list in the event the user only wants to run the task across a subset of DCs.

[adutra]

Hmm I see. What about:

apiVersion: control.k8ssandra.io/v1alpha1
kind: K8ssandraClusterTask
metadata:
  name: upgradesstables
spec:
  cluster:
    name: cluster1
    namespace: k8ssandra-operator
  datacenters: # a list of target DC names, assume all DCs in spec order if not provided
    - dc3
    - dc1
  jobs:
    - name: upgradesstables-dc1
      command: upgradesstables

[olim7t]

Will jobs reuse the CassandraJob / CassandraCommand resources from cass-operator, or do we plan to have a different set of commands?

[burmanm]

Hmm I see. What about:

Would listing datacenters restrict the output to only those, and if left without datacenters array - run to all datacenters?

.. and reading the comment, it seems so ;)

[jsanda]

Will jobs reuse the CassandraJob / CassandraCommand resources from cass-operator, or do we plan to have a different set of commands?

To help answer this question I think it would be good to look at examples for a few different types of CassandraTask to see how they look at the cluster level. I can provide some examples if you would like.

It would also be good to discuss what the status will look like.

[adutra]

Will jobs reuse the CassandraJob / CassandraCommand resources from cass-operator, or do we plan to have a different set of commands?

Also, I was a bit concerned about the shape that JobArguments is taking: it lists all arguments to all known commands, but as the number of supported commands grow, I feat that this struct will become a bag of unrelated stuff and could eventually become easy to be used in wrong ways. @burmanm do you have the same feeling? What would you suggest to improve this?

[burmanm]

@adutra So I guess the alternative would look like:

jobs:
  rebuild:
    - name: "run-my-upgradesstable"
      source_datacenter: "dc1"

Or did you have some other struct in mind?

[jsanda]

@adejanovski Can we please create an issue for documenting cluster this feature and make sure it gets picked up after #739 is merged?