Replies: 1 comment 2 replies
-
I agree with the goal, but I think the proposed solution is hard to implement. Fundamentally, Armada has to understand all objects submitted to it, including the implications of scheduling them, e.g., the resources consumed by each CRD considered for scheduling. Recall that MCAD is purpose-built for Spark (I don't think it even does pods?). As I've suggested before, I recommend chatting with Sam re this. He has an idea based on hijacking the K8s scheduler that could achieve this goal without us having to manually implement support for every CRD we want to support. |
Beta Was this translation helpful? Give feedback.
2 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Motivation
Users want to be able to submit and monitor various types of Kubernetes objects. The Kubernetes community has started pushing for the concept of an “Operator”, which is a third-party custom resource definition with a corresponding controller. The community of these projects maintains and supports best practices for running this objects. We should consider how to design Armada to take third party objects.
There are way too many third party objects to support and GR/GR-OSS may not have experts on staff to enable Spark, Dask, Ray, MPI, etc on Kubernetes so we should turn to open source solutions to enable more use cases of Armada.
Community CRDs of Interest to Armada/GR
All of these have active community support (>50 contributors) and are being actively developed. These are not small projects and we should not try to replicate this with a Pod first API.
Kubeflow is in the process of migrating all their training operators to a single operator but the original implementation had each operator as a separate controller.
Issues on our repo:
Community Impact:
Batch-Processing-Gateway from Apple (https://github.com/apple/batch-processing-gateway) demonstrates how to run Spark jobs across multiple kubernetes cluster. However, this is a common pattern from many other communities that want to take these objects and run them in a multi-cluster way. Argo-Workflows are working on multi-cluster solution (argoproj/argo-workflows#3523)
In general, we should be thinking of ways to bring other batch-compute frameworks into the multi-cluster world.
High Level User Diagram
Sketch of Implementation:
Our Armada API can take a general object and it is sent to a Kubernetes cluster that has the corresponding controller deployed. I would think you would want the arbitrary object and a corresponding tag that says what type of object this is so that our executors could understand this object and our scheduler can send this object to the Kubernetes cluster that has the capability of running this object.
MCAD (https://github.com/IBM/multi-cluster-app-dispatcher) provides a nice idea of this in practice. They released a blog post talking about how to enable Ray and MCAD: https://www.anyscale.com/blog/gang-scheduling-ray-clusters-on-kubernetes-with-multi-cluster-app-dispatcher
Beta Was this translation helpful? Give feedback.
All reactions