Prepare 0.46.1 release (#2039)

* Prepare 0.46.1 release

* Auto-update of E2E template

---------

Co-authored-by: GitHub Actions <actions@github.com>

README.md

@@ -89,7 +89,7 @@
     <a href="https://www.zenml.io/company#team">Meet the Team</a>
     <br />
     <br />
-    🎉 Version 0.46.0 is out. Check out the release notes
+    🎉 Version 0.46.1 is out. Check out the release notes
     <a href="https://github.com/zenml-io/zenml/releases">here</a>.
     <br />
     <br />

RELEASE_NOTES.md

@@ -1,4 +1,38 @@
 <!-- markdown-link-check-disable -->
+# 0.46.1
+
+The 0.46.1 release introduces support for Service Accounts and API Keys that
+can be used to authenticate with the ZenML server from environments that do not
+support the web login flow, such as CI/CD environments, for example.
+
+Also included in this release are some documentation updates and bug fixes,
+notably moving the database migration logic deployed with the Helm chart out of
+the init containers and into a Kubernetes Job, which makes it possible to scale
+out the ZenML server deployments without the risk of running into database
+migration conflicts.
+
+## What's Changed
+* Small improvements to Hub docs page by @strickvl in https://github.com/zenml-io/zenml/pull/2015
+* Pin OpenAI integration to `<1.0.0` by @strickvl in https://github.com/zenml-io/zenml/pull/2027
+* Make error message nicer for when two artifacts that share a prefix are found by @strickvl in https://github.com/zenml-io/zenml/pull/2023
+* Move db-migration to `job` instead of `init-container` to allow replicas by @safoinme in https://github.com/zenml-io/zenml/pull/2021
+* Fix stuck/broken CI by @strickvl in https://github.com/zenml-io/zenml/pull/2032
+* Increase `step.source_code` Cut-Off Limit by @fa9r in https://github.com/zenml-io/zenml/pull/2025
+* Improve artifact linkage logging in MCP by @avishniakov in https://github.com/zenml-io/zenml/pull/2016
+* Upgrade feast so apidocs don't fail no mo by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2028
+* Remove NumPy Visualizations for 2D Arrays by @fa9r in https://github.com/zenml-io/zenml/pull/2033
+* Fix user activation bug by @stefannica in https://github.com/zenml-io/zenml/pull/2037
+* Remove `create_new_model_version` arg of `ModelConfig` by @avishniakov in https://github.com/zenml-io/zenml/pull/2030
+* Extend the wait period in between PyPi package publication and Docker image building for releases by @strickvl in https://github.com/zenml-io/zenml/pull/2029
+* Make `zenml up` prefill username when launching dashboard by @strickvl in https://github.com/zenml-io/zenml/pull/2024
+* Add warning when artifact store cannot be loaded by @strickvl in https://github.com/zenml-io/zenml/pull/2011
+* Add extra config to `Kaniko` docs by @safoinme in https://github.com/zenml-io/zenml/pull/2019
+* ZenML API Keys and Service Accounts by @stefannica in https://github.com/zenml-io/zenml/pull/1840
+
+
+**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.46.0..0.46.1
+
+
 # 0.46.0
 
 This release brings some upgrades, documentation updates and bug fixes. Notably,

examples/e2e/README.md

@@ -81,19 +81,21 @@ This template uses
 to demonstrate how to perform major critical steps for Continuous Training (CT)
 and Continuous Delivery (CD).
 
-It consists of two pipelines with the following high-level setup:
+It consists of three pipelines with the following high-level setup:
 <p align="center">
-  <img height=300 src=".assets/00_pipelines_composition.png">
+  <img height=800 src=".assets/00_pipelines_composition.png">
 </p>
 
-Both pipelines are inside a shared Model Control Plane model context - training pipeline creates and promotes new Model Control Plane version and inference pipeline is reading from inference Model Control Plane version. This makes those pipelines closely connected, while ensuring that only quality-assured Model Control Plane versions are used to produce predictions delivered to stakeholders.
+All pipelines are leveraging the Model Control Plane to bring all parts together - the training pipeline creates and promotes a new Model Control Plane version with a trained model object in it, deployment pipeline uses the inference Model Control Plane version (the one promoted during training) to create a deployment service and inference pipeline using deployment service from the inference Model Control Plane version and store back new set of predictions as a versioned data artifact for future use. This makes those pipelines closely connected while ensuring that only quality-assured Model Control Plane versions are used to produce predictions delivered to stakeholders.
 * [CT] Training
   * Load, split, and preprocess the training dataset
   * Search for an optimal model object architecture and tune its hyperparameters
   * Train the model object and evaluate its performance on the holdout set
   * Compare a recently trained model object with one promoted earlier
   * If a recently trained model object performs better - stage it as a new inference model object in model registry
   * On success of the current model object - stage newly created Model Control Plane version as the one used for inference
+* [CD] Deployment
+  * Deploy a new prediction service based on the model object connected to the inference Model Control Plane version.
 * [CD] Batch Inference
   * Load the inference dataset and preprocess it reusing object fitted during training
   * Perform data drift analysis reusing training dataset of the inference Model Control Plane version as a reference
@@ -119,23 +121,27 @@ The project loosely follows [the recommended ZenML project structure](https://do
 
 ```
 .
-├── pipelines               # `zenml.pipeline` implementations
-│   ├── batch_inference.py  # [CD] Batch Inference pipeline
-│   └── training.py         # [CT] Training Pipeline
-├── steps                   # logically grouped `zenml.steps` implementations
-│   ├── alerts              # alert developer on pipeline status
-│   ├── data_quality        # quality gates built on top of drift report
-│   ├── etl                 # ETL logic for dataset
-│   ├── hp_tuning           # tune hyperparameters and model architectures
-│   ├── inference           # inference on top of the model from the registry
-│   ├── promotion           # find if a newly trained model will be new inference
-│   └── training            # train and evaluate model
-├── utils                   # helper functions
+├── configs                   # pipelines configuration files
+│   ├── deployer_config.yaml  # the configuration of the deployment pipeline
+│   ├── inference_config.yaml # the configuration of the batch inference pipeline
+│   └── train_config.yaml     # the configuration of the training pipeline
+├── pipelines                 # `zenml.pipeline` implementations
+│   ├── batch_inference.py    # [CD] Batch Inference pipeline
+│   ├── deployment.py         # [CD] Deployment pipeline
+│   └── training.py           # [CT] Training Pipeline
+├── steps                     # logically grouped `zenml.steps` implementations
+│   ├── alerts                # alert developer on pipeline status
+│   ├── deployment            # deploy trained model objects
+│   ├── data_quality          # quality gates built on top of drift report
+│   ├── etl                   # ETL logic for dataset
+│   ├── hp_tuning             # tune hyperparameters and model architectures
+│   ├── inference             # inference on top of the model from the registry
+│   ├── promotion             # find if a newly trained model will be new inference
+│   └── training              # train and evaluate model
+├── utils                     # helper functions
 ├── .dockerignore
-├── inference_config.yaml   # the configuration of the batch inference pipeline
-├── Makefile                # helper scripts for quick start with integrations
-├── README.md               # this file
-├── requirements.txt        # extra Python dependencies 
-├── run.py                  # CLI tool to run pipelines on ZenML Stack
-└── train_config.yaml       # the configuration of the training pipeline
+├── Makefile                  # helper scripts for quick start with integrations
+├── README.md                 # this file
+├── requirements.txt          # extra Python dependencies 
+└── run.py                    # CLI tool to run pipelines on ZenML Stack
 ```

examples/e2e/configs/deployer_config.yaml

@@ -0,0 +1,44 @@
+# Apache Software License 2.0
+# 
+# Copyright (c) ZenML GmbH 2023. All rights reserved.
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+# http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# 
+
+# environment configuration
+settings:
+  docker:
+    required_integrations:
+      - aws
+      - evidently
+      - kubeflow
+      - kubernetes
+      - mlflow
+      - sklearn
+      - slack
+
+# configuration of steps  
+steps:
+  notify_on_success:
+    parameters:
+      notify_on_success: False
+
+# configuration of the Model Control Plane
+model_config:
+  name: e2e_use_case
+  version: staging
+
+# pipeline level extra configurations
+extra:
+  notify_on_failure: True
+

examples/e2e/configs/inference_config.yaml

@@ -0,0 +1,44 @@
+# Apache Software License 2.0
+# 
+# Copyright (c) ZenML GmbH 2023. All rights reserved.
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+# http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# 
+
+# environment configuration
+settings:
+  docker:
+    required_integrations:
+      - aws
+      - evidently
+      - kubeflow
+      - kubernetes
+      - mlflow
+      - sklearn
+      - slack
+
+# configuration of steps  
+steps:
+  notify_on_success:
+    parameters:
+      notify_on_success: False
+
+# configuration of the Model Control Plane
+model_config:
+  name: e2e_use_case
+  version: staging
+
+# pipeline level extra configurations
+extra:
+  notify_on_failure: True
+

examples/e2e/configs/train_config.yaml

@@ -0,0 +1,112 @@
+# Apache Software License 2.0
+# 
+# Copyright (c) ZenML GmbH 2023. All rights reserved.
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+# http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# 
+
+# environment configuration
+settings:
+  docker:
+    required_integrations:
+      - aws
+      - evidently
+      - kubeflow
+      - kubernetes
+      - mlflow
+      - sklearn
+      - slack
+
+# configuration of steps  
+steps:
+  model_trainer:
+    parameters:
+      name: e2e_use_case
+  compute_performance_metrics_on_current_data:
+    parameters:
+      target_env: staging
+  promote_with_metric_compare:
+    parameters:
+      mlflow_model_name: e2e_use_case
+      target_env: staging
+  notify_on_success:
+    parameters:
+      notify_on_success: False
+
+# configuration of the Model Control Plane
+model_config:
+  name: e2e_use_case
+  license: apache
+  description: e2e_use_case E2E Batch Use Case
+  audience: All ZenML users
+  use_cases: |
+    The ZenML E2E project project demonstrates how the most important steps of 
+    the ML Production Lifecycle can be implemented in a reusable way remaining 
+    agnostic to the underlying infrastructure, and shows how to integrate them together 
+    into pipelines for Training and Batch Inference purposes.
+  ethics: No impact.
+  tags:
+  - e2e
+  - batch
+  - sklearn
+  - from template
+  - ZenML delivered
+  create_new_model_version: true
+
+# pipeline level extra configurations
+extra:
+  notify_on_failure: True
+  # This set contains all the model configurations that you want 
+  # to evaluate during hyperparameter tuning stage.
+  model_search_space:
+    random_forest:
+      model_package: sklearn.ensemble
+      model_class: RandomForestClassifier
+      search_grid:
+        criterion:
+          - gini
+          - entropy
+        max_depth:
+          - 2
+          - 4
+          - 6
+          - 8
+          - 10
+          - 12
+        min_samples_leaf:
+          range:
+            start: 1
+            end: 10
+        n_estimators:
+          range:
+            start: 50
+            end: 500
+            step: 25
+    decision_tree:
+      model_package: sklearn.tree
+      model_class: DecisionTreeClassifier
+      search_grid:
+        criterion:
+          - gini
+          - entropy
+        max_depth:
+          - 2
+          - 4
+          - 6
+          - 8
+          - 10
+          - 12
+        min_samples_leaf:
+          range:
+            start: 1
+            end: 10

examples/e2e/pipelines/__init__.py

@@ -18,3 +18,4 @@
 
 from .batch_inference import e2e_use_case_batch_inference
 from .training import e2e_use_case_training
+from .deployment import e2e_use_case_deployment

examples/e2e/pipelines/batch_inference.py

@@ -15,7 +15,6 @@
 # limitations under the License.
 #
 
-
 from steps import (
     data_loader,
     drift_quality_gate,
@@ -25,13 +24,10 @@
     notify_on_success,
 )
 
-from zenml import get_pipeline_context, pipeline
+from zenml import pipeline
 from zenml.artifacts.external_artifact import ExternalArtifact
 from zenml.integrations.evidently.metrics import EvidentlyMetricConfig
 from zenml.integrations.evidently.steps import evidently_report_step
-from zenml.integrations.mlflow.steps.mlflow_deployer import (
-    mlflow_model_registry_deployer_step,
-)
 from zenml.logger import get_logger
 
 logger = get_logger(__name__)
@@ -49,7 +45,13 @@ def e2e_use_case_batch_inference():
     # Link all the steps together by calling them and passing the output
     # of one step as the input of the next step.
     ########## ETL stage  ##########
-    df_inference, target = data_loader(is_inference=True)
+    df_inference, target, _ = data_loader(
+        random_state=ExternalArtifact(
+            model_artifact_pipeline_name="e2e_use_case_training",
+            model_artifact_name="random_state",
+        ),
+        is_inference=True,
+    )
     df_inference = inference_data_preprocessor(
         dataset_inf=df_inference,
         preprocess_pipeline=ExternalArtifact(
@@ -70,15 +72,7 @@ def e2e_use_case_batch_inference():
     )
     drift_quality_gate(report)
     ########## Inference stage  ##########
-    deployment_service = mlflow_model_registry_deployer_step(
-        registry_model_name=get_pipeline_context().extra["mlflow_model_name"],
-        registry_model_version=ExternalArtifact(
-            model_artifact_name="promoted_version",
-        ),
-        replace_existing=True,
-    )
     inference_predict(
-        deployment_service=deployment_service,
         dataset_inf=df_inference,
         after=["drift_quality_gate"],
     )