Chnage behaviour of using step based resources

docs/book/stacks-and-components/component-guide/orchestrators/skypilot-vm.md

@@ -384,7 +384,11 @@ One of the key features of the SkyPilot VM Orchestrator is the ability to run ea
 
 The SkyPilot VM Orchestrator allows you to configure resources for each step individually. This means you can specify different VM types, CPU and memory requirements, and even use spot instances for certain steps while using on-demand instances for others.
 
-In order to enable this, you will need to update your orchestrator configuration to use `configure_step_resources=True` or create a new orchestrator with this setting enabled. This setting allows the orchestrator to configure resources for each step individually. Once your orchestrator is configured to allow step-specific resources, you can pass a `SkypilotBaseOrchestratorSettings` object to the `settings` parameter of the `@step` decorator. This object allows you to define various attributes such as `instance_type`, `cpus`, `memory`, `use_spot`, `region`, and more.
+By default, the orchestrator will use the resources specified in the orchestrator settings for each step and make sure that the VMs are provisioned with the appropriate resources. However, you can disable this behavior by setting the `disable_step_based_settings` parameter to `True` in the orchestrator configuration. You can do this using the following command:
+
+```shell
+zenml orchestrator update <ORCHESTRATOR_NAME> --disable_step_based_settings=True
+```
 
 Here's an example of how to configure specific resources for a step for the AWS cloud:
 

src/zenml/integrations/skypilot/flavors/skypilot_orchestrator_base_vm_config.py

@@ -116,14 +116,13 @@ class SkypilotBaseOrchestratorConfig(  # type: ignore[misc] # https://github.com
     """Skypilot orchestrator base config.
 
     Attributes:
-        configure_step_resources: Enables the orchestrator to run configured steps.
-            This will be used to determine whether to run the entire pipeline
-            in one single VM or to run each step in a separate VM if the
-            orchestrator is configured to run steps separately with some custom
-            resources.
+        disable_step_based_settings: whether to disable step-based settings.
+            If True, the orchestrator will run all steps with the pipeline
+            settings in one single VM. If False, the orchestrator will run
+            each step with its own settings in separate VMs if provided.
     """
 
-    configure_step_resources: bool = False
+    disable_step_based_settings: bool = False
 
     @property
     def is_local(self) -> bool:

src/zenml/integrations/skypilot/orchestrators/skypilot_base_vm_orchestrator.py

@@ -200,20 +200,39 @@ def prepare_or_run_pipeline(
                 deployment=deployment, step_name=pipeline_step_name
             )
 
-        if self.config.configure_step_resources:
+        if self.config.disable_step_based_settings:
+            # Run the entire pipeline in one VM
+            command = PipelineEntrypointConfiguration.get_entrypoint_command()
+            args = PipelineEntrypointConfiguration.get_entrypoint_arguments(
+                deployment_id=deployment.id
+            )
+        else:
+            for step_name, step in deployment.step_configurations.items():
+                step_settings = cast(
+                    SkypilotBaseOrchestratorSettings,
+                    self.get_settings(step),
+                )
+                if step_settings != settings:
+                    logger.info(
+                        "At least one step has different settings than the "
+                        "pipeline. The step with different settings will be "
+                        "run in a separate VM.\n"
+                        "You can configure the orchestrator to disable this "
+                        "behavior by updating the `disable_step_based_settings` "
+                        "in your orchestrator configuration."
+                        "By running the following command: "
+                        "`zenml orchestrator update --disable-step-based-settings=True`"
+                    )
+                    break
             # Run each step in a separate VM if configured.
             # Build entrypoint command and args for the orchestrator VM.
-            # This will internally also build the command/args for all step pods.
+            # This will internally also build the command/args for all step VMs.
             command = SkypilotOrchestratorEntrypointConfiguration.get_entrypoint_command()
             args = SkypilotOrchestratorEntrypointConfiguration.get_entrypoint_arguments(
                 run_name=orchestrator_run_name,
                 deployment_id=deployment.id,
             )
-        else:
-            command = PipelineEntrypointConfiguration.get_entrypoint_command()
-            args = PipelineEntrypointConfiguration.get_entrypoint_arguments(
-                deployment_id=deployment.id
-            )
+
         entrypoint_str = " ".join(command)
         arguments_str = " ".join(args)