Skypilot with Kubernetes #3033

safoinme · 2024-09-22T01:16:42Z

Describe changes

I implemented/fixed _ to achieve _.

Pre-requisites

Please ensure you have done the following:

I have read the CONTRIBUTING.md document.
If my change requires a change to docs, I have updated the documentation accordingly.
I have added tests to cover my changes.
I have based my new branch on develop and the open PR is targeting develop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.
If my changes require changes to the dashboard, these changes are communicated/requested.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Other (add details above)

Summary by CodeRabbit

Release Notes

New Features
- Introduced a comprehensive guide for integrating SkyPilot with Kubernetes in the documentation.
- Added the SkypilotKubernetesIntegration to enhance ZenML's integration capabilities.
- Implemented the SkypilotKubernetesOrchestrator for managing pipeline execution in Kubernetes environments.
Documentation
- Expanded documentation detailing installation, configuration, and limitations of the Kubernetes orchestrator.
- Included a code example for setting up Kubernetes orchestrator settings.
Bug Fixes
- Added warnings regarding unsupported features in the Kubernetes orchestrator.

This commit adds the Skypilot Kubernetes integration to enable remote orchestration of ZenML pipelines on VMs. The integration includes the necessary initialization, configuration, and flavor classes. It also adds the Skypilot Kubernetes orchestrator and its settings. This integration provides an alternative to the local orchestrator for running pipelines on Kubernetes.

…method

coderabbitai · 2024-09-22T01:16:57Z

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

The changes introduce a new Kubernetes orchestrator for the SkyPilot integration within the ZenML framework. This includes comprehensive documentation updates, the addition of new classes and constants, and enhancements to existing orchestrator functionalities. The new orchestrator facilitates the execution of ZenML pipelines on virtual machines in Kubernetes environments, with specific configurations and settings tailored for this deployment model.

Changes

Files	Change Summary
`docs/book/component-guide/orchestrators/skypilot-vm.md`	Added a section for Kubernetes orchestrator documentation, including installation, configuration, limitations, and examples.
`src/zenml/integrations/__init__.py`	Introduced `SkypilotKubernetesIntegration` to the ZenML integrations module.
`src/zenml/integrations/constants.py`	Added constant `SKYPILOT_KUBERNETES` for SkyPilot Kubernetes integration.
`src/zenml/integrations/skypilot/orchestrators/skypilot_base_vm_orchestrator.py`	Modified `prepare_or_run_pipeline` to enhance command execution and environment variable handling for Kubernetes.
`src/zenml/integrations/skypilot_kubernetes/__init__.py`	Created `SkypilotKubernetesIntegration` class with attributes and installation checks for the integration.
`src/zenml/integrations/skypilot_kubernetes/flavors/__init__.py`	Introduced module for Skypilot Kubernetes orchestrator flavors, importing necessary classes.
`src/zenml/integrations/skypilot_kubernetes/flavors/skypilot_orchestrator_kubernetes_vm_flavor.py`	Defined classes for orchestrator settings, configuration, and flavor specific to Kubernetes.
`src/zenml/integrations/skypilot_kubernetes/orchestrators/__init__.py`	Initialized Skypilot Kubernetes orchestrators, specifying public interface.
`src/zenml/integrations/skypilot_kubernetes/orchestrators/skypilot_kubernetes_vm_orchestrator.py`	Implemented `SkypilotKubernetesOrchestrator` class with methods for managing Kubernetes pipeline execution.

Possibly related PRs

Databricks integrations #2823: Introduces a new orchestrator for Databricks, enhancing orchestration capabilities similar to the Kubernetes orchestrator.
One-click GCP stack deployments #2833: Focuses on GCP stack deployments, which may relate to the Kubernetes orchestrator's deployment processes within ZenML.

Poem

🐰 In the meadow, changes bloom,
Kubernetes joins the orchestration room.
Pipelines dance on virtual air,
With SkyPilot, we have flair!
New settings, flavors, all in line,
ZenML's future, oh so fine! 🌼

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
-- I pushed a fix in commit <commit_id>, please review it.
-- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
-- @coderabbitai generate unit testing code for this file.
-- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
-- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
-- @coderabbitai read src/utils.ts and generate unit testing code.
-- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
-- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

…otKubernetesOrchestrator prepare_environment_variable method

htahir1

In addition to my comments, we also need to add docs :-)

src/zenml/integrations/skypilot/orchestrators/skypilot_base_vm_orchestrator.py

htahir1 · 2024-09-22T07:38:37Z

src/zenml/integrations/skypilot/orchestrators/skypilot_base_vm_orchestrator.py

-                "DOCKER_USERNAME": docker_username,
-                "DOCKER_PASSWORD": docker_password,
-            }
+            task_envs["DOCKER_USERNAME"] = (docker_username,)


why is this a tuple now?

it's not I haven't pushed the latest stuff

htahir1 · 2024-09-22T07:38:54Z

src/zenml/integrations/skypilot/orchestrators/skypilot_base_vm_orchestrator.py


        # Run the entire pipeline

        # Set the service connector AWS profile ENV variable
        self.prepare_environment_variable(set=True)

        try:
+            if isinstance(self.cloud, sky.clouds.Kubernetes):
+                image = image
+                run_command = f"/opt/venv/bin/{entrypoint_str} {arguments_str}"


this seems like a weird entrypoint.. have you tested it?

Now this is a tricky one, Not sure what would be the exact solution long term but if we give skypilot the entrypoint it would use the default Python which is Miniconada but we want to use the one we have already set when building docker image in pipeline run and that python version is in /opt/venv/bin/

then i would at lest make it configurable from the outside in case something weird changes

Not just something weird, but anyone using a custom image which does not magically use the same venv.
What is skypilot doing exactly? Because just calling python should just use the venv/system python that's active, or do they do some magic to prevent that?

They actually do some magic which basically forces us to either figure out which env to use or stop building the image as it is and use skypilot image to build on top of it and consider it like lightning studio where we only run command afterward, @htahir1 @michael-zenml which means stop building docker image for this integration and have pip install and zenml install a commands that get's passed to skypilot setup argument

src/zenml/integrations/skypilot/orchestrators/skypilot_base_vm_orchestrator.py

src/zenml/integrations/skypilot_kubernetes/orchestrators/skypilot_kubernetes_vm_orchestrator.py

...zenml/integrations/skypilot_kubernetes/flavors/skypilot_orchestrator_kubernetes_vm_flavor.py

…otKubernetesOrchestrator prepare_environment_variable method

src/zenml/integrations/skypilot/orchestrators/skypilot_base_vm_orchestrator.py

stefannica · 2024-09-23T08:31:14Z

src/zenml/integrations/skypilot/orchestrators/skypilot_base_vm_orchestrator.py


        # Run the entire pipeline

        # Set the service connector AWS profile ENV variable
        self.prepare_environment_variable(set=True)

        try:
+            if isinstance(self.cloud, sky.clouds.Kubernetes):


This might seem mostly cosmetic, but I'll say it anyway: you're adding implementation specific conditions in the base class, which is supposed to be implementation agnostic. The correct way to do this is through static attributes, properties or methods that the implementations can override.

Well i think i want to re-work the whole Skypilot integration if possible so instead of have a base abstraction and many sub integration which blocks one of the most important features of skypilot cross-cloud/env failover, tho I totally agree with your comment

src/zenml/integrations/skypilot_kubernetes/orchestrators/skypilot_kubernetes_vm_orchestrator.py

…_orchestrator.py Co-authored-by: Stefan Nica <stefan@zenml.io>

…l-io/zenml into feature/skypilot-kubernetes

…nector

…l-io/zenml into feature/skypilot-kubernetes

…ent path for Kubernetes

schustmi · 2024-09-24T12:55:40Z

src/zenml/integrations/skypilot/orchestrators/skypilot_base_vm_orchestrator.py


        # Run the entire pipeline

        # Set the service connector AWS profile ENV variable
        self.prepare_environment_variable(set=True)

        try:
+            if isinstance(self.cloud, sky.clouds.Kubernetes):
+                run_command = f"$VIRTUAL_ENV{entrypoint_str} {arguments_str}"


I think this doesn't work right, $VIRTUAL_ENV is for example /opt/venv and the python is in /opt/venv/bin/python. So this needs the bin directory to work, and then also a fallback in case $VIRTUAL_ENV is not set.

And we need to make sure this is actually working

yes this is important @safoinme

schustmi · 2024-09-24T12:56:46Z

src/zenml/integrations/skypilot_kubernetes/orchestrators/skypilot_kubernetes_vm_orchestrator.py

+        Raises:
+            ValueError: If no service connector is found.
+        """
+        connector = self.get_connector()


This seems like a misuse of the method, this isn't actually preparing any environment variables.

Maybe this should be in SkypilotKubernetesOrchestrator.prepare_pipeline_deployment(...)? This method gets called for each stack component before a pipeline run

htahir1 · 2024-09-24T13:07:49Z

@coderabbitai summary

coderabbitai · 2024-09-24T13:07:55Z

Actions performed

Summary regeneration triggered.

htahir1

Approved barring michaels comment

…l-io/zenml into feature/skypilot-kubernetes

schustmi · 2024-09-24T16:10:06Z

src/zenml/integrations/skypilot/orchestrators/skypilot_base_vm_orchestrator.py


        # Run the entire pipeline

        # Set the service connector AWS profile ENV variable
        self.prepare_environment_variable(set=True)

        try:
+            if isinstance(self.cloud, sky.clouds.Kubernetes):
+                run_command = f"${{VIRTUAL_ENV:+$VIRTUAL_ENV/bin/}}{entrypoint_str} {arguments_str}"


Which shell should this run in? Doesn't work for me somehow

Ah it's the double brackets because of the f-string, my bad. If you tested this and it works, feel free to merge.

…l-io/zenml into feature/skypilot-kubernetes

safoinme added 5 commits September 21, 2024 14:07

Add Skypilot Kubernetes integration for remote orchestration

342fa34

Refactor SkypilotKubernetesOrchestrator prepare_environment_variable …

7590584

…method

Refactor SkypilotKubernetesOrchestrator prepare_environment_variable …

03951d8

…method

Refactor SkypilotKubernetesOrchestrator prepare_environment_variable …

89718f0

…method

github-actions bot added internal To filter out internal PRs and issues enhancement New feature or request labels Sep 22, 2024

safoinme added 2 commits September 22, 2024 02:20

Refactor SkypilotBaseOrchestrator setup_credentials method

775253f

Refactor SkypilotBaseOrchestrator setup_credentials method and Skypil…

4b42940

…otKubernetesOrchestrator prepare_environment_variable method

htahir1 requested changes Sep 22, 2024

View reviewed changes

Refactor SkypilotBaseOrchestrator setup_credentials method and Skypil…

305322e

…otKubernetesOrchestrator prepare_environment_variable method

safoinme added the run-slow-ci label Sep 22, 2024

stefannica reviewed Sep 23, 2024

View reviewed changes

safoinme and others added 7 commits September 23, 2024 16:13

Update src/zenml/integrations/skypilot/orchestrators/skypilot_base_vm…

edae0e8

…_orchestrator.py Co-authored-by: Stefan Nica <stefan@zenml.io>

Merge branch 'develop' into feature/skypilot-kubernetes

2c68fa6

Merge branch 'feature/skypilot-kubernetes' of https://github.com/zenm…

38922b2

…l-io/zenml into feature/skypilot-kubernetes

Merge branch 'develop' into feature/skypilot-kubernetes

364f8ac

Refactor SkypilotKubernetesOrchestrator to handle missing service con…

85e7f63

…nector

Merge branch 'feature/skypilot-kubernetes' of https://github.com/zenm…

2c421b0

…l-io/zenml into feature/skypilot-kubernetes

Refactor SkypilotBaseOrchestrator to use the correct virtual environm…

f4639f6

…ent path for Kubernetes

safoinme requested review from schustmi and htahir1 September 24, 2024 12:45

safoinme and others added 2 commits September 24, 2024 13:47

specll check

46194cc

Merge branch 'develop' into feature/skypilot-kubernetes

abf58c4

safoinme requested a review from stefannica September 24, 2024 12:48

schustmi requested changes Sep 24, 2024

View reviewed changes

htahir1 approved these changes Sep 24, 2024

View reviewed changes

safoinme added 3 commits September 24, 2024 16:54

fix run command

b3a4520

Merge branch 'feature/skypilot-kubernetes' of https://github.com/zenm…

1eef87a

…l-io/zenml into feature/skypilot-kubernetes

Refactor SkypilotKubernetesVMOrchestrator to remove unnecessary code

615c083

safoinme requested a review from schustmi September 24, 2024 16:00

Merge branch 'develop' into feature/skypilot-kubernetes

5b48068

schustmi requested changes Sep 24, 2024

View reviewed changes

safoinme added 2 commits September 24, 2024 17:29

Refactor SkypilotKubernetesVMOrchestrator to remove unnecessary code

98973a6

Merge branch 'feature/skypilot-kubernetes' of https://github.com/zenm…

126eefb

…l-io/zenml into feature/skypilot-kubernetes

safoinme requested review from schustmi and htahir1 September 25, 2024 07:02

schustmi approved these changes Sep 25, 2024

View reviewed changes

bcdurak approved these changes Sep 25, 2024

View reviewed changes

safoinme merged commit b711963 into develop Sep 25, 2024
79 of 83 checks passed

safoinme deleted the feature/skypilot-kubernetes branch September 25, 2024 07:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Skypilot with Kubernetes #3033

Skypilot with Kubernetes #3033

safoinme commented Sep 22, 2024 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Sep 22, 2024 •

edited

Loading

Review skipped

Walkthrough

Changes

Possibly related PRs

Poem

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

Documentation and Community

htahir1 left a comment

htahir1 Sep 22, 2024

safoinme Sep 22, 2024

htahir1 Sep 22, 2024

safoinme Sep 22, 2024

htahir1 Sep 22, 2024

schustmi Sep 23, 2024

safoinme Sep 23, 2024 •

edited

Loading

stefannica Sep 23, 2024

safoinme Sep 23, 2024

schustmi Sep 24, 2024

htahir1 Sep 24, 2024

schustmi Sep 24, 2024

htahir1 commented Sep 24, 2024

coderabbitai bot commented Sep 24, 2024

htahir1 left a comment

schustmi Sep 24, 2024

schustmi Sep 24, 2024

Skypilot with Kubernetes #3033

Skypilot with Kubernetes #3033

Conversation

safoinme commented Sep 22, 2024 • edited by coderabbitai bot Loading

Describe changes

Pre-requisites

Types of changes

Summary by CodeRabbit

Release Notes

coderabbitai bot commented Sep 22, 2024 • edited Loading

Review skipped

Walkthrough

Changes

Possibly related PRs

Poem

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

Documentation and Community

htahir1 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

safoinme Sep 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

htahir1 commented Sep 24, 2024

coderabbitai bot commented Sep 24, 2024

htahir1 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

safoinme commented Sep 22, 2024 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Sep 22, 2024 •

edited

Loading

safoinme Sep 23, 2024 •

edited

Loading