Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: remove Keptn scheduler #3826

Merged
merged 7 commits into from
Nov 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 0 additions & 20 deletions .github/actions/spelling/expect.txt
Original file line number Diff line number Diff line change
Expand Up @@ -111,9 +111,6 @@ crd
crdkustomizeresource
crdkustomizewebhookpatch
crt
csidrivers
csinodes
csistoragecapacities
ctrlclient
ctrlcommon
ctx
Expand Down Expand Up @@ -157,7 +154,6 @@ dupl
dynamicinformer
dynatrace
ecr
elasticquota
elif
ENABLEGITINFO
endblock
Expand Down Expand Up @@ -254,7 +250,6 @@ ICertificate
IConfig
IEvaluation
IEvent
iface
ifdef
ifeq
ifile
Expand Down Expand Up @@ -364,7 +359,6 @@ keptnworkloadversionstatus
kics
klakegg
klc
klcpermit
klog
Klogr
klt
Expand All @@ -374,7 +368,6 @@ kubebuilder
kubeconform
kubeflow
kubescape
kubescheduler
kubeutils
kustomization
kustomize
Expand Down Expand Up @@ -477,7 +470,6 @@ otelprom
otelsdk
otelurl
otlp
otlpgrpc
otlptrace
otlptracegrpc
OUTDIR
Expand All @@ -487,16 +479,12 @@ oyaml
PARAMSx
PDTR
perc
persistentvolumeclaims
persistentvolumes
phaseitem
phasetraceid
pid
pipefail
pmig
poc
poddisruptionbudgets
podgroups
podtato
podtatohead
podtatometric
Expand Down Expand Up @@ -537,7 +525,6 @@ redis
refcache
remediations
replicasets
replicationcontrollers
resourcereference
revisionhistorylimit
Rexed
Expand All @@ -563,15 +550,13 @@ sclient
screenshot
sdkmetric
sdktest
sdktrace
secureparameters
selfsigned
semconv
sendserviceaccount
serrors
serviceaccount
serviceaccountspec
serviceapp
serviceport
setuptools
Shandilya
Expand Down Expand Up @@ -602,12 +587,10 @@ stakeholders
statefulset
statefultest
stdouttrace
storageclasses
storageversion
sts
subchart
subfolders
subjectaccessreviews
substitue
substr
Sudipto
Expand Down Expand Up @@ -643,7 +626,6 @@ timespan
timezone
tmp
tocstop
tokenreviews
toml
totalscore
tplvalues
Expand All @@ -652,7 +634,6 @@ traceparent
tracerfactory
tracetest
trivy
trivyignore
trunc
ttlsecondsafterfinished
ttlsecondstask
Expand All @@ -669,7 +650,6 @@ Utkarsh
validatingwebhookconfigurations
vanalysis
vanalysisdefinition
vcluster
Vec
versioned
versioning
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/CI.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ env:
GO_VERSION: "~1.23"
# renovate: datasource=github-releases depName=kubernetes-sigs/controller-tools
CONTROLLER_TOOLS_VERSION: "v0.16.5"
ENVTEST_K8S_VERSION: "1.24.2"
ENVTEST_K8S_VERSION: "1.27.0"
defaults:
run:
shell: bash
Expand Down
8 changes: 1 addition & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,13 +93,7 @@ You can find our roadmap [here](https://github.com/orgs/keptn/projects/10).
## Installation

Keptn can be installed on any Kubernetes cluster
running Kubernetes >=1.24.

For users running [vCluster](https://www.vcluster.com/),
please note that you may need to modify
your configuration before installing Keptn; see
[Running Keptn with vCluster](https://keptn.sh/stable/docs/installation/configuration/vcluster/)
for more information.
running Kubernetes >=1.27.

Use the following command sequence
to install the latest release of Keptn:
Expand Down
2 changes: 1 addition & 1 deletion chart/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ annotations:
artifacthub.io/license: Apache-2.0
artifacthub.io/operator: 'true'
artifacthub.io/operatorCapabilities: Full Lifecycle
kubeVersion: '>= 1.24.0-0'
kubeVersion: '>= 1.27.0-0'
type: application
version: 0.8.0
appVersion: v2.3.0 # x-release-please-version
Expand Down
6 changes: 0 additions & 6 deletions docs/docs/components/assets/scheduler.yaml

This file was deleted.

1 change: 0 additions & 1 deletion docs/docs/components/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ Keptn consists of the following main components:

* [Keptn Lifecycle Operator](./lifecycle-operator/index.md)
* [Keptn Metrics Operator](./metrics-operator.md)
* [Keptn Scheduler](./scheduling.md)
* [Keptn Certificate Manager](./certificate-operator.md)

The architectural diagram:
Expand Down
5 changes: 0 additions & 5 deletions docs/docs/components/lifecycle-operator/deployment-flow.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,11 +34,6 @@ and [CloudEvents](https://cloudevents.io/)
are emitted at each phase
to provide additional Observability of the execution flow.

Keptn implements a
[Permit Scheduler Plugin](https://kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework/#permit)
that blocks the binding of the pods to a node
until all the pre-conditions are fulfilled.

A Kubernetes deployment is started by the deployment engine
that is implemented
(such as Flux or Argo)
Expand Down
6 changes: 3 additions & 3 deletions docs/docs/components/lifecycle-operator/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,12 @@ transitions between different stages such as development, testing, staging, and
graph TD;

G((CRs + manifests)) -- apply --> H[Kubernetes API]
I[Scheduler] <-- schedules Pods --> H
J[Lifecycle controllers] <-- orchestrates --> H
I[Lifecycle controller] <-- orchestrates --> H
I <-- schedules Pods --> H


style H fill:#006bb8,stroke:#fff,stroke-width:px,color:#fff
style I fill:#d8e6f4,stroke:#fff,stroke-width:px,color:#006bb8
style J fill:#d8e6f4,stroke:#fff,stroke-width:px,color:#006bb8
style G fill:#fff,stroke:#123,stroke-width:px,color:#006bb8
```

Expand Down
2 changes: 1 addition & 1 deletion docs/docs/components/lifecycle-operator/keptn-apps.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ Implementing Keptn applications provides the following benefits:
of all workloads together rather than individually.
* You can define pre-deployment evaluations and tasks
that must all complete successfully
before the scheduler binds the pods to the nodes.
before the Kubernetes scheduler binds the pods to the nodes.
For information about how to disable the blocking
functionality, please refer to
[this section](./keptn-non-blocking.md#keptn-non-blocking-deployment-functionality).
Expand Down
5 changes: 2 additions & 3 deletions docs/docs/components/lifecycle-operator/webhook.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,8 @@ metadata:
```

The mutating webhook only modifies specifically annotated resources in the annotated namespace.
When the webhook receives a request for a new pod, it first either replaces the scheduler
with the Keptn Scheduler, or adds the
[Scheduling Gate](https://keptn.sh/stable/docs/components/scheduling/#keptn-scheduling-gates-for-k8s-127-and-above).
When the webhook receives a request for a new pod, it adds the
[Scheduling Gate](https://keptn.sh/stable/docs/components/scheduling/#keptn-scheduling-gates).

In the next step it looks for the workload annotations:

Expand Down
77 changes: 4 additions & 73 deletions docs/docs/components/scheduling.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,27 +6,12 @@ comments: true

Keptn integrates with Kubernetes scheduling to block
the deployment of applications that do not satisfy Keptn defined pre-deployment checks.
The default scheduling paradigm is different
depending on the version of Kubernetes you are running:

* On Kubernetes versions 1.26 and older,
Keptn uses the **Keptn Scheduler** to block application deployment when appropriate
and orchestrate the deployment process.

* On Kubernetes version 1.27 and greater,
scheduling is implemented using
[Kubernetes scheduling gates](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-scheduling-readiness/).

These two implementations are discussed below.

## Keptn Scheduling Gates for K8s 1.27 and above

When Keptn is running on Kubernetes version 1.27 and greater
and the Keptn Helm value `lifecycleOperator.schedulingGatesEnabled` is set to `true`,
Keptn uses the
[Pod Scheduling Readiness K8s API](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-scheduling-readiness)
The default scheduling paradigm is implemented using
[Kubernetes scheduling gates](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-scheduling-readiness/)
to gate Pods until the required deployment checks pass.

## Keptn Scheduling Gates

When a workload is applied to a Kubernetes cluster,
the Mutating Webhook checks each Pod for annotations.
If
Expand Down Expand Up @@ -55,60 +40,6 @@ if the Pod is updated, it is not gated again:
{% include "./assets/gate-removed.yaml" %}
```

## Keptn Scheduler for K8s 1.26 and earlier

The **Keptn Scheduler** works by registering itself as a Permit plugin within the Kubernetes
scheduling cycle that ensures that Pods are scheduled to a node until and unless the
pre-deployment checks have finished successfully.
This helps to prevent Pods from being bound to nodes that are not yet ready for them,
which can lead to errors and downtime.
Furthermore, it also allows users to control the deployment of an application based on
customized rules that can take into consideration more parameters than what the default
scheduler has (typically CPU and memory values).

The Keptn Scheduler uses the Kubernetes
[Scheduler Framework](https://kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework/) and is based on the
[Scheduler Plugins Repository](https://github.com/kubernetes-sigs/scheduler-plugins/tree/master).
Additionally, it registers itself as
a [Permit plugin](https://kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework/#permit).

### How does the Keptn Scheduler works

Firstly the Mutating Webhook checks for annotations on Pods to see if it is annotated with
[Keptn specific annotations](../guides/integrate.md#basic-annotations).
If the annotations are present, the Webhook assigns the **Keptn Scheduler** to the Pod.
This ensures that the Keptn Scheduler only manages Pods that have been annotated for Keptn.
A Pod `test-pod` modified by the Mutating Webhook looks as follows:

```yaml
{% include "./assets/scheduler.yaml" %}
```

If the Pod is annotated with Keptn specific annotations, the Keptn Scheduler retrieves
the WorkloadVersion CRD that is associated with the Pod.
The **WorkloadVersion CR** contains information about the `pre-deployment` checks that
need to be performed before the Pod can be scheduled.

The Keptn Scheduler then checks the status of the WorkloadVersion CR to see
if the `pre-deployment` checks have finished successfully.
If the pre-deployment checks have finished successfully, the **Keptn Scheduler** allows
the Pod to be bound to a node.
If the `pre-deployment` checks have not finished successfully within 5 minutes,
the Pod is not bound to a node and it remains in a Pending state.

> **Note**
The blocking behavior can be changed by configuring non-blocking deployment
functionality.
More information can be found in the
[Keptn non-blocking deployment section](./lifecycle-operator/keptn-non-blocking.md).

It is important to note that the Keptn Scheduler is a plugin to the default Kubernetes scheduler.
This means that all of the checks that are by default performed by the default Kubernetes scheduler
will also be performed by the **Keptn Scheduler**.
For example, if there is not enough capacity on any node to schedule the Pod,
the Keptn Scheduler will not be able to schedule it, even if the `pre-deployment`
checks have finished successfully.

## Integrating Keptn with your custom scheduler

Keptn scheduling logics are compatible with
Expand Down
2 changes: 0 additions & 2 deletions docs/docs/contribute/general/release-checklist.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,6 @@ This page should be used to document the current process to create a full releas
1. release metrics operator
1. bump metrics operator chart version in charts repo and merge chart release PR
1. bump metrics operator chart version in main chart
1. release scheduler
1. fix scheduler pr conflicts re-generate lifecycle-operator Helm chart docs
1. release lifecycle operator
1. bump lifecycle operator chart version in charts repo and merge chart release PR
1. bump lifecycle-operator chart version in main chart
Expand Down
4 changes: 1 addition & 3 deletions docs/docs/contribute/software/dev-environ.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@ To prepare to contribute to the Keptn project, we recommend that you:

* [lifecycle-operator](https://github.com/keptn/lifecycle-toolkit/tree/main/lifecycle-operator)
* [metrics-operator](https://github.com/keptn/lifecycle-toolkit/tree/main/metrics-operator)
* [scheduler](https://github.com/keptn/lifecycle-toolkit/tree/main/scheduler)
* [keptn-cert-manager](https://github.com/keptn/lifecycle-toolkit/tree/main/keptn-cert-manager)

Each of these is described in the
Expand Down Expand Up @@ -51,7 +50,6 @@ documentation:
* `metrics-operator`
* `lifecycle-operatory`
* `cert-manager`
* Keptn `scheduler`

At the top level of the repository,
you also see the `runtimes` directory.
Expand Down Expand Up @@ -180,7 +178,7 @@ From the `lifecycle-operator` directory, run the component tests:

### End-to-end test

From the `lifecycle-operator` or `scheduler` directory, run the end-to-end tests:
From the `lifecycle-operator` directory, run the end-to-end tests:

```shell
make e2e-test
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/getting-started/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ The following is an imperative walkthrough.
- [Docker](https://docs.docker.com/get-started/overview/)
- [kubectl](https://kubernetes.io/docs/reference/kubectl/)
- [Helm](https://helm.sh/docs/intro/install/)
- A Kubernetes cluster >= 1.24 (we recommend [Kubernetes kind](https://kind.sigs.k8s.io/docs/user/quick-start/))
- A Kubernetes cluster >= 1.27 (we recommend [Kubernetes kind](https://kind.sigs.k8s.io/docs/user/quick-start/))
(`kind create cluster`)

## Objectives
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/guides/assets/multi-stage-delivery/chart.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
apiVersion: v2
name: podtato-head
kubeVersion: ">= 1.24.0-0"
kubeVersion: ">= 1.27.0-0"
type: application
version: 0.1.0
appVersion: "v0.1.0"
4 changes: 2 additions & 2 deletions docs/docs/guides/otel.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,12 +136,12 @@ kubectl edit configmap otel-collector-conf \
```

When the `otel-collector` pod is up and running,
restart the `keptn-scheduler` (if installed) and `lifecycle-operator`
restart the `lifecycle-operator`
so they can pick up the new configuration:

```shell
kubectl rollout restart deployment \
-n keptn-system keptn-scheduler lifecycle-operator
-n keptn-system lifecycle-operator
```

Keptn begins to collect OpenTelemetry metrics
Expand Down
Loading
Loading