[WIP] k8s 1.31 #2004

The CBOR decode and "appendix a" unit tests cover specific serialization behaviors that were known to be incomplete at the time. Now that all of those cases have been addressed, the mechanism for skipping those tests can be removed.

apimachinery: API Emulation Versioning

…xpression-evaluation Validate CRD stored messageExpressions with the correct CEL environment

… limit"

…ionPolicy-GA promote PDBUnhealthyPodEvictionPolicy to GA

Remove test dependency on swagger.json to fix client-go tests

This is a feature of the underlying k8s.io/klog/v2/ktesting which is useful also when using the Kubernetes ktesting.

The DRA plugin does that. It didn't actually work and only printed an error message about NodeInfo not implementing klog.KMetata. That's not a compile-time check due to limitations with Go generics and had been missed earlier.

Signed-off-by: cpanato <ctadeu@gmail.com>

KEP-4222: Enable JSON-compatible base64 encoding of []byte for CBOR.

…am-consistent-read-from-cache-supports-pagination Revert "apiserver/storage/cacher: consistent read from cache supports limit"

…ogging scheduler: fix klog.KObjSlice when applied to []*NodeInfo

…etcdserver

Remove 5 seconds from TestWatchStreamSeparation execution by reusing etcdserver

Signed-off-by: googs1025 <googs1025@gmail.com>

…ListFromCache

The underlying CBOR library will by default encode a value to and from byte string if its type implements encoding.BinaryMarshaler or encoding.BinaryUnmarshaler, respectively. This is now disabled via an option to avoid diverging from JSON in those cases.

update publishing rules to use go1.22.4 for all branches

…tests-decrease-time apiserver/storage: decrease running time of RunWatchSemantics

This reverts commit 82e9ce7.

This reverts commit d65029f.

This reverts commit 0455544.

This reverts commit a3b7cb9.

controlplane/apiserver: Trim managedFields off self-requested informers

kubernetes#112809 tightened what regular expressions are allowed and now requires that they start matching with a double dash.

Signed-off-by: Nilekh Chaudhari <1626598+nilekhc@users.noreply.github.com>

pkg/printers: drop deprecated pointer package

drop deprecated PollWithContext and adopt PollUntilContextTimeout ins…

Sample cli plugin help

…-controlplane Generic Control Plane: Add Extra.DisableAvailableConditionController for kube-aggregator

Revert "Update opentelemetry dependencies to the latest release."

… apiserver. Signed-off-by: Siyuan Zhang <sizhang@google.com>

node: memory manager: fix the metrics tests

…ymarshaler KEP-4222: Disable recognition of Binary(Unm|M)arshaler in CBOR serializer.

DRA: log output

picks up fix for data-race in Ping

…netes#125117) * kubectl: add internalTrafficPolicy to Service describer * kubectl: add loadBalancer ipMode to Service describer * kubectl: fix duplicate IP fields in Service describer For a LoadBalancer Service, there were two "IP" fields in the output of `kubectl describe service` if its loadBalancerIP is not empty, which looks ambiguous.

…cs_flake Fix test flakes for TestWatchSemantics

KEP-3857: promote RecursiveReadOnlyMounts feature to beta

pod-resize-scheduler-tests: fix ci-kubernetes-e2e-ec2

…r-watch-list-kcm-units kube-controller-manager: refactor watchlist unit tests

[Storage Version Migrator] fix: increases timeout on CRD update check

bump github.com/moby/spdystream to v0.3.0

…kips KEP-4222: Remove temporary mechanism for skipping CBOR tests.

Ignore reported goroutine leak during SPDY shutdown

Add KUBE_EMULATED_VERSION env variable to set the emulated-version of apiserver

Signed-off-by: joey <zchengjoey@gmail.com>

Signed-off-by: googs1025 <googs1025@gmail.com>

kubeadm: implement ControlPlaneKubeletLocalMode

…-cors-fix hack/local-up-cluster.sh: adapt to stricter --cors-allowed-origins

fix base64 invalid argument when running gce util.sh

Add OWNERS to test/fuzz.

The arguments should contain named parameters, which is now the case. Signed-off-by: Sascha Grunert <sgrunert@redhat.com>

…Count

update kube-network-policies to v0.4.0

…troller_log_fix1 added resourceclaim_controller log info

kubeadm: update the long description about 'kubeadm config' command

…ion' to include the checked resource kind

…-test-wait-until-fresh storage/cacher/cacher_whitebox_test.go: deflake TestWaitUntilFreshAndListFromCache

…ervice Make CRI service interface function arguments consistent

ServiceCIDRs are protected by finalizers and the CIDRs fields are inmutable once set, only the readiness state impact the allocator as it can only allocate IPs if any of the ServiceCIDR is ready. The Add/Update events triggers a reconcilation of the current state of the ServiceCIDR present in the informers with the existing IP allocators. The Delete events are handled directly to update or delete the corresponing IP allocator.

The current results with 100 works and 15k services on a (n2-standard-48) vCPU: 48 RAM: 192 GB are: Old allocator: perf_test.go:139: [RESULT] Duration 1m9.646167533s: [quantile:0.5 value:0.462886801 quantile:0.9 value:0.496662838 quantile:0.99 value:0.725845905] New allocator: perf_test.go:139: [RESULT] Duration 2m12.900694343s: [quantile:0.5 value:0.481814448 quantile:0.9 value:1.3867615469999999 quantile:0.99 value:1.888190671] The new allocator has higher latency but in contrast allow to use a larger number of services, when tested with 65k Services the old allocator etcd crashes with storage exceeded. The scenario is also not realistic, as a continuous and high load on Service creation is not expected.

MultiCIDRServiceAllocator implements a new ClusterIP allocator based on IPAddress object to solve the problems and limitations caused by existing bitmap allocators. However, during the rollout of new versions, deployments need to support a skew of one version between kube-apiservers. To avoid the possible problem where there are multiple Services requests on the skewed apiservers and that both allocate the same IP to different Services, the new allocator will implement a dual-write strategy under the feature gate DisableAllocatorDualWrite. After the MultiCIDRServiceAllocator is GA, the DisableAllocatorDualWrite can be enabled safely as all apiservers will run with the new allocators. The graduation of DisableAllocatorDualWrite can also be used to clean up the opaque API object that contains the old bitmaps. If MultiCIDRServiceAllocator is enabled and DisableAllocatorDualWrite is disable and is a new environment, there is no bitmap object created, hence, the apiserver will initialize it to be able to write on it.

…-run-test-list apiserver/storage: storagetesting.RunTestList validates RemainingItemCount

There was a race caused by having to update claim finalizer and status in two different operations: - Resource claim controller removes allocation, does not yet get to remove the finalizer. - Scheduler prepares an allocation, without adding the finalizer because it's there. - Controller removes finalizer. - Scheduler adds allocation. This is an invalid state. Automatic checking found this during the execution of the "with translated parameters on single node.*supports sharing a claim sequentially" E2E test, but only when run stand-alone. When running in parallel (as in the CI), the bad outcome of the race did not occur. The fix is to check that the finalizer is still set when adding the allocation. The apiserver doesn't check that because it doesn't know which finalizer goes with the allocation result. It could check for "some finalizer", but that is not guaranteed to be correct (could be some unrelated one). Checking the finalizer can only be done with a JSON patch. Despite the complications, having the ability to add multiple pods concurrently to ReservedFor seems worth it (avoids expensive rescheduling or a local retry loop). The resource claim controller doesn't need this, it can do a normal update which implicitly checks ResourceVersion.

The JSON patch approach works, but it is complex. A retry loop is easier to understand (detect conflict, get new claim, try again). There is one additional API call (the get), but in practice this scenario is unlikely.

Signed-off-by: Dr. Stefan Schimanski <stefan.schimanski@gmail.com>

Signed-off-by: Davanum Srinivas <davanum@gmail.com>

…portedVersion-20240627 kubeadm: improve the error/warning messages of `validateSupportedVersion` to include the checked resource kind

This makes the API nicer: resourceClaims: - name: with-template resourceClaimTemplateName: test-inline-claim-template - name: with-claim resourceClaimName: test-shared-claim Previously, this was: resourceClaims: - name: with-template source: resourceClaimTemplateName: test-inline-claim-template - name: with-claim source: resourceClaimName: test-shared-claim A more long-term benefit is that other, future alternatives might not make sense under the "source" umbrella. This is a breaking change. It's justified because DRA is still alpha and will have several other API breaks in 1.31.

Signed-off-by: Davanum Srinivas <davanum@gmail.com>

…ated_volumes Use +default for deprecated volume types

Prune explicit nulls from client-side apply create

Move public key serviceaccount getter to interface, filter by key id

…ack-consistent-read-from-cache-supports-pagination apiserver/storage/cacher: consistent read from cache supports limit

…-to-v0.4.0 Update moby/spdystream to v0.4.0

…core-group code-generator/client-gen: decouple core group from package name 'api'

Signed-off-by: Vinayak Goyal <vinaygo@google.com>

kube-proxy config update

KEP-4633: Only allow anonymous auth for configured endpoints.

…-v0.55.0 Bump `prometheus/common` to v0.55.0

…or_ut chore: add DefaultSelector method ut

Signed-off-by: kerthcet <kerthcet@gmail.com>

KEP-1880 Multiple Service CIDRs: Graduate to Beta (1/2)

Currently, there are some unit tests that are failing on Windows due to various reasons: - IPVS proxy mode is not supported on Windows. - pkg/kubelet/cri/remote was moved to cri-client.

…d when 'upgrade diff' specified a config file

…fault

…' and '--scheduler-manifest' as deprecated flags

unittests: Fixes unit tests for Windows (part 12)

Signed-off-by: Harshal Patil <harpatil@redhat.com>

…-fix DRA: fix scheduler/resource claim controller race

…240628 kubeadm: fix a bug where the path of the manifest can not be specified when 'upgrade diff' specified a config file

KEP-4569: Separate cgroup v1 and v2 manager implementations

) * fix: enable bool-compare rule from testifylint linter Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com> * Update hack/golangci.yaml.in Co-authored-by: Patrick Ohly <patrick.ohly@intel.com> * Update golangci.yaml.in * Update golangci-strict.yaml * Update golangci.yaml.in * Update golangci.yaml.in * Update golangci.yaml.in * Update golangci.yaml.in * Update golangci.yaml * Update golangci-hints.yaml * Update golangci-strict.yaml * Update golangci.yaml.in * Update golangci.yaml * Update mux_test.go --------- Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com> Co-authored-by: Patrick Ohly <patrick.ohly@intel.com>

When replacing the deterministic ResourceClaim name with a generated one this particular piece of the original validation was incorrectly left in place. It's not required anymore that "<pod name>-<claim name in pod spec>" is a valid ResourceClaim name.

Co-Authored-By: Iceber Gu <wei.cai-nat@daocloud.io> Signed-off-by: Davanum Srinivas <davanum@gmail.com>

DRA: remove "source" indirection from v1 Pod API

Remove kubelet soft admission

…in-kubelet clean typos logs in kubelet.

Fix E2E error message

…tes#124946) * feat: add func for generating register code * refactor:remove unused local variable * fix: make the function name singular Signed-off-by: Lin Yang <reaver@flomesh.io> * fix: precisely matching the comment tag for register-gen Signed-off-by: Lin Yang <reaver@flomesh.io> --------- Signed-off-by: Lin Yang <reaver@flomesh.io>

…mission improve logging of pod admission denied

kubeadm: Validate only the first cert entry when external ca mode is used

Signed-off-by: bzsuni <bingzhe.sun@daocloud.io>

…ated add a test case with gated pod-affinity pods to scheduler_perf

…server_logs_api kube-apiserver: default --enable-logs-handler flag to false

* Pod terminationGracePeriodSeconds is always valid Validation of a pod spec will always use the pod's TerminationGracePeriodSeconds value. A set of pod test-helpers have been created to help construct Pods. * remove unused func * reduction * reduce 2 * simplify test * report invalid grace period * update SupplementalGroupPolicy tests

…adm upgrade apply/plan'

Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>

kubeadm: the '--feature-gates' flag is deprecated and no-op for 'kubeadm upgrade apply/plan'

KEP-1880 Multiple Service CIDRs: Graduate to Beta (2/2)

cleaning up some tests after MakePod/MakePodSpec were introduced

fix: enable and fixes rules from testifylint on test package

Signed-off-by: kerthcet <kerthcet@gmail.com>

fix flaky integration test about multi profiles

…eWithContext Use PollUntilContextTimeout to replace PollImmediateWithContext

…-watch-list-to-beta Promote WatchList feature to Beta

Signed-off-by: kerthcet <kerthcet@gmail.com>

…failure Log the error margin to avoid failures in schedule_perf

OpenShift-Rebase-Source: 29eea3c

OpenShift-Rebase-Source: 5a2488c

UPSTREAM: <carry>: filter out RBR and SCC paths from OpenAPI UPSTREAM: <carry>: filter out RBR and SCC paths from OpenAPI Revise as per openshift/kubernetes-apiserver#12 OpenShift-Rebase-Source: 26005f1

UPSTREAM: <carry>: prevent apiservice registration by CRD controller when delegating UPSTREAM: <carry>: prevent CRD registration from fighting with APIServices UPSTREAM: <carry>: always delegate namespaced resources OpenShift-Rebase-Source: d4cd0ba

…en it exists OpenShift-Rebase-Source: 1a1d469

…trap SDN when SDN is down UPSTREAM: <carry>: use hardcoded rest mapper from library-go OpenShift-Rebase-Source: a00f75d

Extend the NodeLogQuery feature to support oc adm node-logs options: - Default NodeLogQuery feature gate to true - Add support for --since, --until, --case-sensitive, --output, options UPSTREAM: <carry>: Extend NodeLogQuery feature Fix handling of the "until" parameter when generating the journalctl command. This was incorrectly being passed with the "since" value.

…signer to token controller :100644 100644 b32534e... 3e694fc... M pkg/controller/serviceaccount/tokens_controller.go OpenShift-Rebase-Source: 891b28f

…ontroller-manager UPSTREAM: <carry>: (squash) kube-controller-manager: allow running bare kube-controller-manager UPSTREAM: <carry>: kube-controller-manager: allow running bare kube-controller-manager UPSTREAM: <carry>: (squash) remove egressnetworkpolicies from gc ignored resources egressnetworkpolicies should not be in garbage collector ignored resources, so users can delete them using "--cascade=foreground" flag. Signed-off-by: Flavio Fernandes <flaviof@redhat.com> OpenShift-Rebase-Source: 6c1dee4 UPSTREAM: <carry>: (squash) kube-controller-manager: allow running bare kube-controller-manager UPSTREAM: <carry>: kube-controller-manager: allow running bare kube-controller-manager UPSTREAM: <carry>: kube-controller-manager: allow running bare kube-controller-manager Fix garbage-collection for CRDs. These types are backed by a CRD and not by openshift-apiserver anymore. DefaultGarbageCollectionPolicy (Unsupported) does not work with CRDs. The `foregroundDeletion` finalizer was set on these CRD objects which blocks deletion indifinetelly as GC will ignore these resources.

…rces from quota OpenShift-Rebase-Source: 7d2a074

…ly to admission plugin OpenShift-Rebase-Source: dd3aeca

UPSTREAM: <carry>: openshift-kube-apiserver: enabled conversion gen for admission configs UPSTREAM: <carry>: openshift-kube-apiserver/admission: fix featuregates resource name UPSTREAM: <carry>: openshift-kube-apiserver/admission: add missing FeatureSets UPSTREAM: <carry>: openshift-kube-apiserver: use github.com/openshift/apiserver-library-go/pkg/labelselector UPSTREAM: <carry>: openshift authenticator: don't allow old-style tokens UPSTREAM: <carry>: oauth-authn: support sha256 prefixed tokens UPSTREAM: <carry>: oauth-token-authn: switch to sha256~ prefix UPSTREAM: <carry>: oauth-token-authn: add sha256~ support to bootstrap authenticator UPSTREAM: <drop>: remove the openshift authenticator from the apiserver In 4.8, we moved the authenticator to be configured via webhookTokenAuthenticators to an endpoint in the oauth-apiserver, this should now be safe to remove. UPSTREAM: <carry>: set ResourceQuotaValidationOptions to true When PodAffinityNamespaceSelector goes to beta or GA this might affect how our ClusterResourceQuota might work UPSTREAM: <carry>: simplify the authorizer patch to allow the flags to function UPSTREAM: <carry>: eliminate unnecessary closure in openshift configuration wiring UPSTREAM: <carry>: add crdvalidation for apiserver.spec.tlsSecurityProfile UPSTREAM: <carry>: openshift-kube-apiserver: Add custom resource validation for network spec UPSTREAM: <carry>: stop overriding flags that are explicitly set UPSTREAM: <carry>: add readyz check for openshift apiserver availability UPSTREAM: <carry>: wait for oauth-apiserver accessibility UPSTREAM: <carry>: provide a new admission plugin to mutate management pods CPUs requests The ManagementCPUOverride admission plugin replaces pod container CPU requests with a new management resource. It applies to all pods that: 1. are in an allowed namespace 2. and have the workload annotation. It also sets the new management resource request and limit and set resource annotation that CRI-O can recognize and apply the relevant changes. For more information, see - openshift/enhancements#703 Conditions for CPUs requests deletion: 1. The namespace should have allowed annotation "workload.openshift.io/allowed": "management" 2. The pod should have management annotation: "workload.openshift.io/management": "{"effect": "PreferredDuringScheduling"}" 3. All nodes under the cluster should have new management resource - "management.workload.openshift.io/cores" 4. The CPU request deletion will not change the pod QoS class UPSTREAM: <carry>: Does not prevent pod creation because of no nodes reason when it runs under the regular cluster Check the `cluster` infrastructure resource status to be sure that we run on top of a SNO cluster and in case if the pod runs on top of regular cluster, exit before node existence check. UPSTREAM: <carry>: do not mutate pods when it has a container with both CPU request and limit Removing the CPU request from the container that has a CPU limit will result in the defaulter to set the CPU request back equals to the CPU limit. UPSTREAM: <carry>: Reject the pod creation when we can not decide the cluster type It is possible a race condition between pod creation and the update of the infrastructure resource status with correct values under Status.ControlPlaneTopology and Status.InfrastructureTopology. UPSTREAM: <carry>: add CRD validation for dnses Add an admission plugin that validates the dnses.operator.openshift.io custom resource. For now, the plugin only validates the DNS pod node-placement parameters. This commit fixes bug 1967745. https://bugzilla.redhat.com/show_bug.cgi?id=1967745 * openshift-kube-apiserver/admission/customresourcevalidation/attributes.go (init): Install operatorv1 into supportedObjectsScheme. * openshift-kube-apiserver/admission/customresourcevalidation/customresourcevalidationregistration/cr_validation_registration.go (AllCustomResourceValidators, RegisterCustomResourceValidation): Register the new plugin. * openshift-kube-apiserver/admission/customresourcevalidation/dns/validate_dns.go: New file. (PluginName): New const. (Register): New function. Register the plugin. (toDNSV1): New function. Convert a runtime object to a versioned DNS. (dnsV1): New type to represent a runtime object that is validated as a versioned DNS. (ValidateCreate, ValidateUpdate, ValidateStatusUpdate): New methods. Implement the ObjectValidator interface, using the validateDNSSpecCreate and validateDNSSpecUpdate helpers. (validateDNSSpecCreate, validateDNSSpecUpdate): New functions. Validate a DNS, using the validateDNSSpec helper. (validateDNSSpec): New function. Validate the spec field of a DNS, using the validateDNSNodePlacement helper. (validateDNSNodePlacement): New function. Validate the node selector and tolerations in a DNS's node-placement parameters, using validateTolerations. (validateTolerations): New function. Validate a slice of corev1.Toleration. * openshift-kube-apiserver/admission/customresourcevalidation/dns/validate_dns_test.go: New file. (TestFailValidateDNSSpec): Verify that validateDNSSpec rejects invalid DNS specs. (TestSucceedValidateDNSSpec): Verify that validateDNSSpec accepts valid DNS specs. * vendor/*: Regenerate. UPSTREAM: <carry>: prevent the kubecontrollermanager service-ca from getting less secure UPSTREAM: <carry>: allow SCC to be disabled on a per-namespace basis UPSTREAM: <carry>: verify required http2 cipher suites In the Apiserver admission, we need to return an error if the required http2 cipher suites are missing from a custom tlsSecurityProfile. Currently, custom cipher suites missing ECDHE_RSA_WITH_AES_128_GCM_SHA256 or ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 result in invalid http2 Server configuration causing the apiservers to crash. See: go/x/net/http2.ConfigureServer for futher information. UPSTREAM: <carry>: drop the warning to use --keep-annotations When a user runs the `oc debug` command for the pod with the management resource, we will inform him that he should pass `--keep-annotations` parameter to the debug command. UPSTREAM: <carry>: admission/managementcpusoverride: cover the roll-back case During the upgrade and roll-back flow 4.7->4.8->4.7, the topology related fields under the infrastructure can be empty because the old API does not support them. The code will equal the empty infrastructure section with the current one. When the status has some other non-empty field, and topology fields are empty, we assume that the cluster currently passes via roll-back and not via the clean install. UPSTREAM: <carry>: Remove pod warning annotation when workload partitioning is disabled UPSTREAM: <carry>: use new access token inactivity timeout field. UPSTREAM: <carry>: apirequestcount validation UPSTREAM: <carry>: Added config node object validation for extreme latency profiles UPSTREAM: <carry>: Add Upstream validation in the DNS admission check patches UPSTREAM: <carry>: Make RestrictedEndpointsAdmission check NotReadyAddresses UPSTREAM: <carry>: Make RestrictedEndpointsAdmission restrict EndpointSlices as well Moved SkipSystemMasterAuthorizers to the authorizer. UPSTREAM: <carry>: Add validation plugin for CRD-based route parity. UPSTREAM: <carry>: Add host assignment plugin for CRD-based routes. UPSTREAM: <carry>: Apply shared defaulters to CRD-based routes. Signed-off-by: Artyom Lukianov <alukiano@redhat.com> Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com> Signed-off-by: Swarup Ghosh <swghosh@redhat.com> OpenShift-Rebase-Source: 932411e OpenShift-Rebase-Source: 1899555 OpenShift-Rebase-Source: 453583e OpenShift-Rebase-Source: bf7e23e UPSTREAM: <carry>: STOR-829: Add CSIInlineVolumeSecurity admission plugin The CSIInlineVolumeSecurity admission plugin inspects inline CSI volumes on pod creation and compares the security.openshift.io/csi-ephemeral-volume-profile label on the CSIDriver object to the pod security profile on the namespace. OpenShift-Rebase-Source: a65c34b UPSTREAM: <carry>: add icsp,idms,itms validation reject creating icsp with idms/itms exist Reject icsp with idms.itms resources exists. According to the discuusion resolution https://docs.google.com/document/d/13h6IJn8wlzXdiPMvCWlMEHOXXqEZ9_GYOl02Wldb3z8/edit?usp=sharing, one of current icsp or new mirror setting crd should be rejected if a user tries to use them on the same cluster. Signed-off-by: Qi Wang <qiwan@redhat.com> UPSTREAM: <carry>: node admission plugin for cpu partitioning The ManagedNode admission plugin makes the Infrastructure.Status.CPUPartitioning field authoritative. This validates that nodes that wish to join the cluster are first configured to properly handle workload pinning For more information see - openshift/enhancements#1213 Signed-off-by: ehila <ehila@redhat.com> UPSTREAM: <carry>: kube-apiserver: allow injection of kube-apiserver options UPSTREAM: <carry>: kube-apiserver: allow rewiring OpenShift-Rebase-Source: 56b49c9 OpenShift-Rebase-Source: bcf574c UPSTREAM: <carry>: openshift-kube-apiserver: add kube-apiserver patches UPSTREAM: <carry>: openshift-kube-apiserver: add kube-apiserver patches

…rver code UPSTREAM: <carry>: openshift-kube-apiserver: enabled conversion gen for admission configs UPSTREAM: <carry>: openshift-kube-apiserver/admission: fix featuregates resource name UPSTREAM: <carry>: openshift-kube-apiserver/admission: add missing FeatureSets UPSTREAM: <carry>: openshift-kube-apiserver: use github.com/openshift/apiserver-library-go/pkg/labelselector UPSTREAM: <carry>: openshift authenticator: don't allow old-style tokens UPSTREAM: <carry>: oauth-authn: support sha256 prefixed tokens UPSTREAM: <carry>: oauth-token-authn: switch to sha256~ prefix UPSTREAM: <carry>: oauth-token-authn: add sha256~ support to bootstrap authenticator UPSTREAM: <drop>: remove the openshift authenticator from the apiserver In 4.8, we moved the authenticator to be configured via webhookTokenAuthenticators to an endpoint in the oauth-apiserver, this should now be safe to remove. UPSTREAM: <carry>: set ResourceQuotaValidationOptions to true When PodAffinityNamespaceSelector goes to beta or GA this might affect how our ClusterResourceQuota might work UPSTREAM: <carry>: simplify the authorizer patch to allow the flags to function UPSTREAM: <carry>: eliminate unnecessary closure in openshift configuration wiring UPSTREAM: <carry>: add crdvalidation for apiserver.spec.tlsSecurityProfile UPSTREAM: <carry>: openshift-kube-apiserver: Add custom resource validation for network spec UPSTREAM: <carry>: stop overriding flags that are explicitly set UPSTREAM: <carry>: add readyz check for openshift apiserver availability UPSTREAM: <carry>: wait for oauth-apiserver accessibility UPSTREAM: <carry>: provide a new admission plugin to mutate management pods CPUs requests The ManagementCPUOverride admission plugin replaces pod container CPU requests with a new management resource. It applies to all pods that: 1. are in an allowed namespace 2. and have the workload annotation. It also sets the new management resource request and limit and set resource annotation that CRI-O can recognize and apply the relevant changes. For more information, see - openshift/enhancements#703 Conditions for CPUs requests deletion: 1. The namespace should have allowed annotation "workload.openshift.io/allowed": "management" 2. The pod should have management annotation: "workload.openshift.io/management": "{"effect": "PreferredDuringScheduling"}" 3. All nodes under the cluster should have new management resource - "management.workload.openshift.io/cores" 4. The CPU request deletion will not change the pod QoS class UPSTREAM: <carry>: Does not prevent pod creation because of no nodes reason when it runs under the regular cluster Check the `cluster` infrastructure resource status to be sure that we run on top of a SNO cluster and in case if the pod runs on top of regular cluster, exit before node existence check. UPSTREAM: <carry>: do not mutate pods when it has a container with both CPU request and limit Removing the CPU request from the container that has a CPU limit will result in the defaulter to set the CPU request back equals to the CPU limit. UPSTREAM: <carry>: Reject the pod creation when we can not decide the cluster type It is possible a race condition between pod creation and the update of the infrastructure resource status with correct values under Status.ControlPlaneTopology and Status.InfrastructureTopology. UPSTREAM: <carry>: add CRD validation for dnses Add an admission plugin that validates the dnses.operator.openshift.io custom resource. For now, the plugin only validates the DNS pod node-placement parameters. This commit fixes bug 1967745. https://bugzilla.redhat.com/show_bug.cgi?id=1967745 * openshift-kube-apiserver/admission/customresourcevalidation/attributes.go (init): Install operatorv1 into supportedObjectsScheme. * openshift-kube-apiserver/admission/customresourcevalidation/customresourcevalidationregistration/cr_validation_registration.go (AllCustomResourceValidators, RegisterCustomResourceValidation): Register the new plugin. * openshift-kube-apiserver/admission/customresourcevalidation/dns/validate_dns.go: New file. (PluginName): New const. (Register): New function. Register the plugin. (toDNSV1): New function. Convert a runtime object to a versioned DNS. (dnsV1): New type to represent a runtime object that is validated as a versioned DNS. (ValidateCreate, ValidateUpdate, ValidateStatusUpdate): New methods. Implement the ObjectValidator interface, using the validateDNSSpecCreate and validateDNSSpecUpdate helpers. (validateDNSSpecCreate, validateDNSSpecUpdate): New functions. Validate a DNS, using the validateDNSSpec helper. (validateDNSSpec): New function. Validate the spec field of a DNS, using the validateDNSNodePlacement helper. (validateDNSNodePlacement): New function. Validate the node selector and tolerations in a DNS's node-placement parameters, using validateTolerations. (validateTolerations): New function. Validate a slice of corev1.Toleration. * openshift-kube-apiserver/admission/customresourcevalidation/dns/validate_dns_test.go: New file. (TestFailValidateDNSSpec): Verify that validateDNSSpec rejects invalid DNS specs. (TestSucceedValidateDNSSpec): Verify that validateDNSSpec accepts valid DNS specs. * vendor/*: Regenerate. UPSTREAM: <carry>: prevent the kubecontrollermanager service-ca from getting less secure UPSTREAM: <carry>: allow SCC to be disabled on a per-namespace basis UPSTREAM: <carry>: verify required http2 cipher suites In the Apiserver admission, we need to return an error if the required http2 cipher suites are missing from a custom tlsSecurityProfile. Currently, custom cipher suites missing ECDHE_RSA_WITH_AES_128_GCM_SHA256 or ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 result in invalid http2 Server configuration causing the apiservers to crash. See: go/x/net/http2.ConfigureServer for futher information. UPSTREAM: <carry>: drop the warning to use --keep-annotations When a user runs the `oc debug` command for the pod with the management resource, we will inform him that he should pass `--keep-annotations` parameter to the debug command. UPSTREAM: <carry>: admission/managementcpusoverride: cover the roll-back case During the upgrade and roll-back flow 4.7->4.8->4.7, the topology related fields under the infrastructure can be empty because the old API does not support them. The code will equal the empty infrastructure section with the current one. When the status has some other non-empty field, and topology fields are empty, we assume that the cluster currently passes via roll-back and not via the clean install. UPSTREAM: <carry>: Remove pod warning annotation when workload partitioning is disabled UPSTREAM: <carry>: use new access token inactivity timeout field. UPSTREAM: <carry>: apirequestcount validation UPSTREAM: <carry>: Added config node object validation for extreme latency profiles UPSTREAM: <carry>: Add Upstream validation in the DNS admission check patches UPSTREAM: <carry>: Make RestrictedEndpointsAdmission check NotReadyAddresses UPSTREAM: <carry>: Make RestrictedEndpointsAdmission restrict EndpointSlices as well Moved SkipSystemMasterAuthorizers to the authorizer. UPSTREAM: <carry>: Add validation plugin for CRD-based route parity. UPSTREAM: <carry>: Add host assignment plugin for CRD-based routes. UPSTREAM: <carry>: Apply shared defaulters to CRD-based routes. Signed-off-by: Artyom Lukianov <alukiano@redhat.com> Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com> Signed-off-by: Swarup Ghosh <swghosh@redhat.com> OpenShift-Rebase-Source: 932411e OpenShift-Rebase-Source: 1899555 OpenShift-Rebase-Source: 453583e OpenShift-Rebase-Source: bf7e23e UPSTREAM: <carry>: STOR-829: Add CSIInlineVolumeSecurity admission plugin The CSIInlineVolumeSecurity admission plugin inspects inline CSI volumes on pod creation and compares the security.openshift.io/csi-ephemeral-volume-profile label on the CSIDriver object to the pod security profile on the namespace. OpenShift-Rebase-Source: a65c34b UPSTREAM: <carry>: add icsp,idms,itms validation reject creating icsp with idms/itms exist Reject icsp with idms.itms resources exists. According to the discuusion resolution https://docs.google.com/document/d/13h6IJn8wlzXdiPMvCWlMEHOXXqEZ9_GYOl02Wldb3z8/edit?usp=sharing, one of current icsp or new mirror setting crd should be rejected if a user tries to use them on the same cluster. UPSTREAM: <carry>: node admission plugin for cpu partitioning The ManagedNode admission plugin makes the Infrastructure.Status.CPUPartitioning field authoritative. This validates that nodes that wish to join the cluster are first configured to properly handle workload pinning For more information see - openshift/enhancements#1213 UPSTREAM: <carry>: kube-apiserver: allow injection of kube-apiserver options UPSTREAM: <carry>: kube-apiserver: allow rewiring OpenShift-Rebase-Source: 56b49c9 OpenShift-Rebase-Source: bcf574c UPSTREAM: <carry>: STOR-1270: Admission plugin to deny deletion of storages.operator.openshift.io UPSTREAM: <carry>: support for both icsp and idms objects Revert: openshift#1310 Add support for ICSP and IDMS objects living at the same time. UPSTREAM: <carry>: openshift-kube-apiserver: add openshift-kube-apisever code UPSTREAM: <carry>: featureset validation moved to CEL UPSTREAM: <carry>: Add context to ObjectValidator TODO: add router validation logic to implement ctx add in ObjectValidator UPSTREAM: <carry>: loosen authentication.spec.type validation UPSTREAM: <carry>: openshift-kube-apiserver: add kube-apiserver patches pod .spec.nodeName should not override project node selector in podNodeEnvironment admission plugin UPSTREAM: <carry>: Fix sets.String and sets.Set[string] type mismatch libray-go uses the genetic Set while upstream still uses the deprecated sets.String in some part of its codes. UPSTREAM: <carry>: Add RouteExternalCertificate validation in Route ObjectValidator UPSTREAM: <carry>: Fix incorrect type casting in admission validate_apiserver UPSTREAM: <carry>: react to library-go changes UPSTREAM: <carry>: Update RouteExternalCertificate validation in Route ObjectValidator

OpenShift-Rebase-Source: 2260f01

…ndler chain UPSTREAM: <carry>: use lifeCycleSignals for isTerminating OpenShift-Rebase-Source: a736659

UPSTREAM: <carry>: apiserver: log new connections during termination UPSTREAM: <carry>: apiserver: create LateConnections events on events in the last 20% of graceful termination time UPSTREAM: <carry>: apiserver: log source in LateConnections event UPSTREAM: <carry>: apiserver: skip local IPs and probes for LateConnections UPSTREAM: <carry>: only create valid LateConnections/GracefulTermination events UPSTREAM: <carry>: kube-apiserver: log non-probe requests before ready UPSTREAM: <carry>: apiserver: create hasBeenReadyCh channel UPSTREAM: <carry>: kube-apiserver: log non-probe requests before ready UPSTREAM: <carry>: kube-apiserver: log non-probe requests before ready UPSTREAM: <carry>: fix termination event(s) validation failures UPSTREAM: <carry>: during the rebase collapse to create termination event it makes recording termination events a non-blocking operation. previously closing delayedStopCh might have been delayed on preserving data in the storage. the delayedStopCh is important as it signals the HTTP server to start the shutdown procedure. it also sets a hard timeout of 3 seconds for the storage layer since we are bypassing the API layer. UPSTREAM: <carry>: rename termination events to use lifecycleSignals OpenShift-Rebase-Source: 15b2d2e

OpenShift-Rebase-Source: 439ec41

OpenShift-Rebase-Source: a137009

OpenShift-Rebase-Source: b9a8eb6

UPSTREAM: <carry>: Remove a redundant output in the tests This line is not necessary for our test usage and should not be an issue in OpenShift (openshift-tests already verifies this correctly). UPSTREAM: <carry>: Remove excessive logging during e2e upgrade test This line makes the upgrade log output unreadable and provides no value during the set of tests it's used in: ``` Jan 12 20:49:25.628: INFO: cluster upgrade is Progressing: Working towards registry.svc.ci.openshift.org/ci-op-jbtg7jjb/release@sha256:144e73d125cce620bdf099be9a85225ade489a95622a70075d264ea3ff79219c: downloading update Jan 12 20:49:26.692: INFO: Poke("http://a74e3476115ce4d2d817a1e5ea608dad-802917831.us-east-1.elb.amazonaws.com:80/echo?msg=hello"): success Jan 12 20:49:28.727: INFO: Poke("http://a74e3476115ce4d2d817a1e5ea608dad-802917831.us-east-1.elb.amazonaws.com:80/echo?msg=hello"): success ``` OpenShift-Rebase-Source: 8e73298

…y running test OpenShift uses these function before any test is run and they cause NPE OpenShift-Rebase-Source: 834af76

… setup UPSTREAM: <carry>: refactor/improve CRD publishing e2e tests in an HA setup OpenShift-Rebase-Source: 51aeef3 OpenShift-Rebase-Source: 7fbb6a4

UPSTREAM: <carry>: Copy hack scripts and tools from openshift/origin UPSTREAM: <carry>: Fix shellcheck failures for copied openshift-hack bash UPSTREAM: <carry>: Enable build, test and verify UPSTREAM: <carry>: Copy README content from origin UPSTREAM: <carry>: Copy watch-termination command from openshift/origin UPSTREAM: <carry>: Switch image and rpm build to golang 1.14 UPSTREAM: <carry>: Copy test annotation from origin UPSTREAM: <carry>: Build openshift-compatible kube e2e binary UPSTREAM: <carry>: Updating openshift-hack/images/hyperkube/Dockerfile.rhel baseimages to mach ocp-build-data config UPSTREAM: <carry>: Update test annotation rules UPSTREAM: <carry>: Enable k8s-e2e-serial UPSTREAM: <carry>: Build with golang 1.15 UPSTREAM: <carry>: (squash) Stop installing recent bash and protoc from source UPSTREAM: <carry>: Add rebase instructions UPSTREAM: <carry>: (squash) Update README.openshift to reflect transition UPSTREAM: <carry>: (squash) Stop annotating origin tests with [Suite:openshift] The detection logic was error-prone (different results based on the repo existing in GOPATH vs not) and whether a test comes from origin can be inferred from the absence of the `[Suite:k8s]` tag. UPSTREAM: <carry>: (squash) Update hyperkube version UPSTREAM: <carry>: (squash) Update OpenShift docs UPSTREAM: <carry>: watch-termination: fix deletion race and write non-graceful message also to termination.log UPSTREAM: <carry>: watch-termination: avoid false positives of NonGracefulTermination events UPSTREAM: <carry>: (squash) remove servicecatalog e2e that was dropped upstream UPSTREAM: <carry>: (squash) Fix annotation rules UPSTREAM: <carry>: (squash) Fix image refs UPSTREAM: <carry>: Updating openshift-enterprise-hyperkube builder & base images to be consistent with ART Reconciling with https://github.com/openshift/ocp-build-data/tree/b0ab44b419faae6b18e639e780a1fa50a1df8521/images/openshift-enterprise-hyperkube.yml UPSTREAM: <carry>: (squash) Retry upstream flakes UPSTREAM: <carry>: (squash) Update test exclussions for 1.20.0 UPSTREAM: <carry>: (squash) Add detail to rebase doc - Add new section 'Maintaining this document' - Move checklist above the instructions to emphasize their importance - Add new section 'Reacting to new commits' - Mention that generated changes in carries should be dropped UPSTREAM: <carry>: Enable CSI snapshot e2e tests All images were uploaded to our quay.io mirror and the tests should succeed. UPSTREAM: <carry>: Stop skipping multi-az test (skipped upstream) UPSTREAM: <carry>: bump tag version & update rebase doc UPSTREAM: <carry>: update rebase doc & image UPSTREAM: <carry>: Add Dockerfile to build pause image Ensuring the target directory exists before writing a file to it. UPSTREAM: <carry>: disable part of hack/verify-typecheck-providerless.sh due to our carry patches UPSTREAM: <carry>: Updating openshift-enterprise-pod images to be consistent with ART Reconciling with https://github.com/openshift/ocp-build-data/tree/691e628254f318ce56efda5edc7448ec743c37b8/images/openshift-enterprise-pod.yml UPSTREAM: <carry>: Updating openshift-enterprise-hyperkube images to be consistent with ART Reconciling with https://github.com/openshift/ocp-build-data/tree/691e628254f318ce56efda5edc7448ec743c37b8/images/openshift-enterprise-hyperkube.yml UPSTREAM: <carry>: Add process overlap detection event to watch-termination NOTE: Squash this to watch-termination commit on rebase. UPSTREAM: <carry>: openshift-hack/images/os/Dockerfile: Add io.openshift.build.versions, etc. For example, consider the current 4.10 RHCOS: $ oc image info -o json registry.ci.openshift.org/ocp/4.10:machine-os-content io.k8s.description: The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly. io.k8s.display-name: Red Hat Universal Base Image 8 io.openshift.build.version-display-names: machine-os=Red Hat Enterprise Linux CoreOS io.openshift.build.versions: machine-os=49.84.202109102026-0 io.openshift.expose-services: io.openshift.tags: base rhel8 A bunch of those seem to be inherited from the UBI base image, so we can leave them alone. But the io.openshift.build.* entries are RHCOS-specific, and are consumed by 'oc adm release new ...' [1,2] and friends to answer questions like "which RHCOS is in this release?": $ oc adm release info -o json quay.io/openshift-release-dev/ocp-release:4.8.12-x86_64 { "kubernetes": { "Version": "1.21.1", "DisplayName": "" }, "machine-os": { "Version": "48.84.202109100857-0", "DisplayName": "Red Hat Enterprise Linux CoreOS" } } Setting this label will avoid failures when consumers like driver-toolkit's version consumer [3]: name: 0.0.1-snapshot-machine-os bump into ci-tools-built machine-os-content images that lack the io.openshift.build.versions declaration of machine-os version [4]: error: unable to create a release: unknown version reference "machine-os" I've gone with generic testing values, so hopefully this is not something that local maintainers need to remember to bump for each OpenShift z stream. [1]: https://github.com/openshift/oc/blob/f94afb52dc8a3185b3b9eacaf92ec34d80f8708d/pkg/cli/admin/release/image_mapper.go#L328-L334 [2]: https://github.com/openshift/oc/blob/f94afb52dc8a3185b3b9eacaf92ec34d80f8708d/pkg/cli/admin/release/annotations.go#L19-L28 [3]: openshift/driver-toolkit@464acca#diff-4caed9b2b966a8fa7a016ae28976634a2d3d1b635c4e820d5c038b2305d6af53R18 [4]: https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_kubernetes/959/pull-ci-openshift-kubernetes-master-images/1438398678602616832#1:build-log.txt%3A97 UPSTREAM: <carry>: update rebase doc UPSTREAM: <carry>: squash with the rest of tooling UPSTREAM: <carry>: Updating openshift-enterprise-pod images to be consistent with ART Reconciling with https://github.com/openshift/ocp-build-data/tree/5b89f5b601508a0bcc0399fd3f34b7aa2e86e90e/images/openshift-enterprise-pod.yml UPSTREAM: <carry>: Updating openshift-enterprise-hyperkube images to be consistent with ART Reconciling with https://github.com/openshift/ocp-build-data/tree/5b89f5b601508a0bcc0399fd3f34b7aa2e86e90e/images/openshift-enterprise-hyperkube.yml UPSTREAM: <carry>: rebase script UPSTREAM: <carry>: Fix networking-related test exclusions Tests that fail on openshift-sdn specifically should be tagged as such, so that they don't also get skipped when running under ovn-kubernetes or third-party network plugins. UPSTREAM: <carry>: Skip "subPath should be able to unmount" NFS test Due to a kernel bug https://bugzilla.redhat.com/show_bug.cgi?id=1854379 in Linux 5.7+ this test fails - the bind-mounted NFS share cannot be cleanly unmounted, gets "Stale file handle" error instead on umount. As a result this test is permafailing on Fedora CoreOS nodes. UPSTREAM: <carry>: Skip GlusterFS tests GlusterFS is not supported in 4.x, we've been running its tests just because we could. Now it does not work on IPv6 systems. E [MSGID: 101075] [common-utils.c:312:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Address family for hostname not supported) UPSTREAM: <carry>: Skip GlusterFS tests The previous commit left two GlusterFS test still running: [sig-storage] Volumes GlusterFS should be mountable [Skipped:ibmcloud] [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] Dynamic Provisioning GlusterDynamicProvisioner should create and delete persistent volumes Skip it, we don't support Gluster and it does not work on ipv6 UPSTREAM: <carry>: 1.22 alpha & other tests disablement UPSTREAM: <carry>: 1.21 alpha & other tests disablement UPSTREAM: <carry>: Enable GenerciEphemeralVolume tests UPSTREAM: <carry>: Re-enable [Feature:NetworkPolicy] tests which were wrongly disabled in rebase UPSTREAM: <carry>: Reenable NetworkPolicy test UPSTREAM: <carry>: Conformance tests (sysctls) should be run We have to run this test for conformance, and the tests pass. Reenable this block which has been disabled for 2 releases (but appears to work fine). UPSTREAM: <carry>: Don't force-disable IPv6, dual-stack, and SCTP tests Instead, openshift-tests will enable or disable them depending on cluster configuration. UPSTREAM: <carry>: update Multi-AZ Cluster Volumes test name This test was renamed upstream in kubernetes@006dc74 UPSTREAM: <carry>: re-enable networking tests after rebase During a bump to k8 ver. 1.22.0, networking tests were disabled to accomplish the bump. This disabled netpol and older network tests. Netpol tests will be enabled in a following PR and therefore only partially fixes BZ. This commit partially fixes bug 1986307. https://bugzilla.redhat.com/show_bug.cgi?id=1986307 UPSTREAM: <drop>: update test annotate rules UPSTREAM: <carry>: Add DOWNSTREAM_OWNERS UPSTREAM: <carry>: clarify downstream approver rules UPSTREAM: <carry>: copy extensions into resulting image UPSTREAM: <carry>: update rebase doc UPSTREAM: <carry>: Fix conformance and serial tests by stopping node cordoning Master nodes already have `master` taint which cannot be tolerated by normal workloads. If we manually cordon the master nodes again, some of the control plane components cannot get rescheduled unless they have toleration to the `node.kubernetes.io/unschedulable` taint. Even if we have the toleration in the pod spec, because of the backwards compability issues scheduler will ignore nodes which have `unschedulable` field set. IOW: - Cordoning master nodes is redundant as masters already have taints - Cordoning master nodes can cause issues which are hard to debug as control-plane components may be evicted/preempted during e2e run(highly unlikely but a possibility). So, let's stop cordoning master nodes. UPSTREAM: <carry>: enable internal traffic policy tests Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1986307 UPSTREAM: <carry>: update rebase doc UPSTREAM: <carry>: enable e2e test after 1.23 rebase in sdn Enable "[sig-network] Conntrack should be able to preserve UDP traffic when initial unready endpoints get ready" after 1.23 rebase in openshift/sdn UPSTREAM: <carry>: Unskip OCP SDN related tests Unskip networkPolicy tests concerning IpBlock and egress rules since both features have now been implemented. UPSTREAM: <carry>: enable should drop INVALID conntrack entries test UPSTREAM: <carry>: update e2es UPSTREAM: revert: <carry>: Unskip OCP SDN related tests These newly-enabled tests are breaking some CI, possibly due to race conditions in the tests. Re-disable them for now. This reverts commit aba8d20. UPSTREAM: <carry>: update hyperkube and image version UPSTREAM: <drop>: disable e2e tests - disable 'ProxyTerminatingEndpoints' feature e2e tests - disable [sig-network] [Feature:Topology Hints] should distribute endpoints evenly see https://bugzilla.redhat.com/show_bug.cgi?id=2079958 for more context UPSTREAM: <carry>: Add kubensenter to the openshift RPM This carry-patch adds the kubensenter script to the openshift-hyperkube RPM, by importing it via the new hack/update-kubensenter.sh script. UPSTREAM: <carry>: Skip session affinity timeout tests in 4.12 and higher the default CNI is OVNKubernetes and these two tests do not pass. Skip them. They are also skipping in the origin test suites for ovnk. UPSTREAM: <carry>: Update kubensenter to use exec instead of direct call Because kubelet relies on systemd's Type=notify mechanism, we don't need or want kubensenter to keep itself in the process tree. exec is best. UPSTREAM: <carry>: update to ginkgo v2 - squash to tooling UPSTREAM: <carry>: update rebase doc UPSTREAM: <carry>: allow annotating with a specific suite If a test specifies a suite, don't append another one to it. We want the ability to add tests to a particular suite without automatically being added to parallel conformance. UPSTREAM: <carry>: Ensure balanced brackets in annotated test names We recently started marking tests with apigroups, and in one case we missed the closing bracket on the annotation resulting in the test being erroneously skipped. This adds a check in the annotation generation, and errors when brackets are unbalanced. ``` Example: $ ./hack/verify-generated.sh FAILURE after 12.870s: hack/verify-generated.sh:13: executing '/home/stbenjam/go/src/github.com/openshift/origin/hack/update-generated.sh' expecting success: the command returned the wrong error code Standard output from the command: Nov 4 14:11:25.026: INFO: Enabling in-tree volume drivers Nov 4 14:11:25.026: INFO: Warning: deprecated ENABLE_STORAGE_GCE_PD_DRIVER used. This will be removed in a future release. Use --enabled-volume-drivers=gcepd instead Nov 4 14:11:25.026: INFO: Enabled gcepd and windows-gcepd in-tree volume drivers Standard error from the command: failed: unbalanced brackets in test name: [Top Level] [sig-scheduling][Early] The openshift-console console pods [apigroup:console.openshift.io should be scheduled on different nodes ^ ``` UPSTREAM: <carry>: add CSI migration feature gates for vSphere and Azure File This commit is the next natural step for commits 2d9a8f9 and d37e84c. It introduces custom feature gates to enable the CSI migration in vSphere and Azure File plugins. See openshift/enhancements#549 for details. Stop <carrying> the patch when CSI migration becomes GA (i.e. features.CSIMigrationAzureFile / features.CSIMigrationVSphere are GA). UPSTREAM: <carry>: Skip in-tree topology tests win Azure Disk migrated to CSI Skip test that depend on in-tree Azure Disk volume plugin that (wrongly) uses failure domains for value of "topology.kubernetes.io/zone" label in Azure regions that don't have availability zones. Our e2e tests blindly use that label and expect that a volume provisioned in such a "zone" can be used only by nodes in that "zone" (= topology domain). This is false, Azure Disk CSI driver can use such a volume in any zone and therefore the test may randomly fail. See https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=2066865 UPSTREAM: <carry>: Stop ignoring generated openapi definitions openshift/origin needs to be able to vendor these definitions so they need to be committed. OpenShift-Rebase-Source: 514f181 OpenShift-Rebase-Source: 87e220b OpenShift-Rebase-Source: b25e156 OpenShift-Rebase-Source: 2256387 OpenShift-Rebase-Source: e4d66c1 OpenShift-Rebase-Source: 5af594b UPSTREAM: <carry>: disable tests for features in alpha UPSTREAM: <carry>: disable tests dependent on StackDriver UPSTREAM: <carry>: add default sysctls for kubelet in rpm UPSTREAM: <carry>: add new approvers UPSTREAM: <carry>: update rebase doc UPSTREAM: <carry>: update hyperkube image version Updated builder as well. UPSTREAM: <carry>: add missing generated file UPSTREAM: <carry>: Add OpenShift tooling, images, configs and docs Add CSI mock volume tests. In upstream these tests were moved to a different package, so we stopped generating their names in OpenShift. This patch fixes that. UPSTREAM: <carry>: Add OpenShift tooling, images, configs and docs Disable CSI mock tests for SELinux and RecoverVolumeExpansionFailure, which are alpha features and require additional work to get enabled. UPSTREAM: <carry>: Add OpenShift tooling, images, configs and docs UPSTREAM: <carry>: update rebase doc UPSTREAM: <carry>: disable failing dnsPolicy test UPSTREAM: <carry>: Create minimal wrapper needed to run k8s e2e tests UPSTREAM: <carry>: Change annotation mechanics to allow injecting testMaps and filter out tests UPSTREAM: <carry>: Move k8s-specific rules to our fork UPSTREAM: <carry>: Create minimal wrapper needed to run k8s e2e tests UPSTREAM: <carry>: Add OpenShift tooling, images, configs and docs Update the list of tests that should be skipped. UPSTREAM: <carry>: Force using host go always and use host libriaries UPSTREAM: <carry>: ignore vendor when generating code UPSTREAM: <carry>: ignore vendor when installing ncpu from hack/tools UPSTREAM: <carry>: move test rules from origin These were brought back in o/o PRs as follows: - netpol - openshift/origin#26775 - schedulerpreemption - openshift/origin#27874 UPSTREAM: <carry>: UserNamespacesSupport feature was rename to UserNamespacesStatelessPodsSupport See commit 531d38e. UPSTREAM: <carry>: allow apiserver-library-go to depend on k8s.io/kubernetes UPSTREAM: <carry>: Add OpenShift tooling, images, configs and docs Remove commitchecker. UPSTREAM: <carry>: Force using host go always and use host libriaries UPSTREAM: <carry>: Add OpenShift tooling, images, configs and docs Update builder images. UPSTREAM: <carry>: Add OpenShift tooling, images, configs and docs Bump builder and base images to OCP 4.15 and RHEL 9 (where possible). UPSTREAM: <carry>: Add OpenShift tooling, images, configs and docs Update REBASE.openshift.md file with new RHEL 9 images. UPSTREAM: <carry>: Add OpenShift tooling, images, configs and docs Remove "git rerere" suggestion. This has shown to be problematic in some cases. UPSTREAM: <carry>: Fix sporadic 141 errors in build-rpms "head" sometimes exits before "rpmspec" finishes piping it all its data. Workaround that by separating the rpmspec and head calls. UPSTREAM: <carry>: Disable e2e tests related to AdmissionWebhookMatchConditions UPSTREAM: <carry>: Add OpenShift tooling, images, configs and docs 1. Fix failure while running the verify.import-boss case 2. Add verify-govulncheck.sh to the excluded pattern This requires a new package to be installed on the fly and the same fails with the following error. `go: golang.org/x/vuln/cmd/govulncheck@v1.0.1: cannot query module due to -mod=vendor` The above error needs to be fixed before enabling this `govulncheck` UPSTREAM: <carry>: switch to go1.21 UPSTREAM: <carry>: use snyk file UPSTREAM: <carry>: RPM: Split apiserver, scheduler, k-c-m, kubelet into subpackages This change should allow us to install a much smaller set of binaries into RHCOS while preserving functional compatibility with with anyone who installs `openshift-hyperkube` today as it requires all sub packages. Those wishing to have just the kubelet can begin installing `openshift-hyperkube-kubelet` -rwxr-xr-x. 2 root root 129M Jan 1 1970 /usr/bin/kube-apiserver -rwxr-xr-x. 2 root root 114M Jan 1 1970 /usr/bin/kube-controller-manager -rwxr-xr-x. 2 root root 54M Jan 1 1970 /usr/bin/kube-scheduler -rwxr-xr-x. 2 root root 105M Jan 1 1970 /usr/bin/kubelet -rwxr-xr-x. 2 root root 3.5K Jan 1 1970 /usr/bin/kubensenter Should save about 297M or 74% in most environments where the kubelet is all that's desired. It's not clear to me why these were ever in the RPM since OCP 4.x but this packaging should remain compatible as openshift-hyperkube depends on - openshift-kubelet - openshift-kube-apiserver - openshift-kube-scheduler - openshift-kube-controller-manager UPSTREAM: <carry>: openshift-hack/images/os: delete All the logic there is geared towards `machine-os-content` which is no longer used at all in the cluster. Nowadays, the container to modify is `rhel-coreos`, which is what is already being done in CI: https://github.com/openshift/release/blob/463a8f244ba0f807e76e6fdf974f98d24efd1ced/ci-operator/config/openshift/kubernetes/openshift-kubernetes-master.yaml#L87-L97 UPSTREAM: <carry>: Disable SCCs in k8s-e2e.test namespaces We want to run upstream e2e tests ignored by SCCs. Make sure the test namespaces have label security.openshift.io/disable-securitycontextconstraints: true and disabled podSecurityLabelSync. UPSTREAM: <carry>: Enable SELinux tests Now that k8s-e2e.test is not affected by SCCs, all SELinux tests should pass. UPSTREAM: <carry>: update test rules UPSTREAM: <carry>: permanently disable NodeLogQuery e2e test Tests require SSH configuration and is part of the parallel suite, which does not create the bastion host. Enabling the test would result in the bastion being created for every parallel test execution. Given that we have existing oc and WMCO tests that cover this functionality, we can safely disable it. UPSTREAM: <carry>: clean OpenShift tooling UPSTREAM: <carry>: Add Dockerfile to buld kube-apiserver for openshift-install architectures UPSTREAM: <carry>: Add OpenShift tooling, images, configs and docs UPSTREAM: <carry>: Create minimal wrapper needed to run k8s e2e tests UPSTREAM: <carry>: Add update go workspace step to the update flow Given we verify go workspace, we need to do `update-go-workspace` step during `make update` UPSTREAM: <carry>: Provide SCC access via RBAC UPSTREAM: <carry>: add native build to installer image This is needed for the s390x/ppc64le arches since we just cross-compile to linux amd/arm64. UPSTREAM: <carry>: update docker image to use go 1.22 UPSTREAM: <carry>: update rules.go UPSTREAM: <carry>: Skip eviction test on tainted nodes for SNO jobs UPSTREAM: <carry>: OCPBUGS-34102: force static build of linux binaries Setting `KUBE_STATIC_OVERRIDES` is necessary for the kubernetes build system to attempt a static build but we also need to set `GO_COMPLIANCE_EXCLUDE` so the `CGO_ENABLED` value is not overridden by the fips-or-die toolchain used to build the release payload. This fixes an issue when running the openshift-installer in centos7/rhel8 systems which fails with: ``` E0521 18:04:24.925722 2077 server.go:317] "unable to start the controlplane" err="unable to run command \"cluster-api/kube-apiserver\" to check for flag \"insecure-port\": exit status 1" logger="controller-runtime.test-env" tries=4 ERROR failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to run cluster api system: failed to run local control plane: unable to start control plane itself: failed to start the controlplane. ret\ ried 5 times: unable to run command "cluster-api/kube-apiserver" to check for flag "insecure-port": exit status 1 ``` because it's trying to run a dynamically-linked kube-apiserver binary.

…lures of important pods UPSTREAM: <carry>: provide unique reason for pod probe event during termination OpenShift-Rebase-Source: 01542fc

OpenShift-Rebase-Source: 6386eb2

…ocalhost to force KS to use localhost set the following flag in kubescheduler (oc edit kubescheduler cluster) unsupportedConfigOverrides: arguments: unsupported-kube-api-over-localhost:: - "true" UPSTREAM: <carry>: allows for switching KS to talk to Kube API over localhost-squash to other This commit is addendum to openshift@04eabe5 to stop using cc and start relying on scheduler config options OpenShift-Rebase-Source: aa9dde2 UPSTREAM: <carry>: allows for switching KS to talk to Kube API over localhost

UPSTREAM: <carry>: management workloads enhancement 741 UPSTREAM: <carry>: lower verbosity of managed workloads logging Support for managed workloads was introduced by PR#627. However, the the CPU manager reconcile loop now seems to flood kubelet log with "reconcileState: skipping pod; pod is managed" warnings. Lower the verbosity of these log messages. UPSTREAM: <carry>: set correctly static pods CPUs when workload partitioning is disabled UPSTREAM: <carry>: Remove reserved CPUs from default set Remove reserved CPUs from default set when workload partitioning is enabled. Co-Authored-By: Brent Rowsell <browsell@redhat.com> Signed-off-by: Artyom Lukianov <alukiano@redhat.com> Signed-off-by: Don Penney <dpenney@redhat.com> OpenShift-Rebase-Source: b762ced OpenShift-Rebase-Source: 63cf793 OpenShift-Rebase-Source: 32af64c UPSTREAM: <carry>: add management support to kubelet UPSTREAM: <carry>: OCPBUGS-29520: fix cpu manager default cpuset check in workload partitioned env (this can be squashed to 04070bb UPSTREAM: : add management support to kubelet) Workload partitioning makes the separation between reserved and workload cpus more strict. It is therefore expected the reserved cpus are NOT part of the default cpuset and the existing check was overzealous. First execution of kubelet after reboot never gets here as the cpuset is computed on line 209. However a kubelet restart without reboot skips this code, recovers from state file and runs the check on line 220. This was uncovered by decoupling the cpu manager state file cleanup from kubelet restart, doing it only once at reboot as part of OCPBUGS-24366 UPSTREAM: <carry>: add management workload check for guaranteed qos when static pods have workload partitioning enabled we should not alter their resources if they are Guaranteed QoS, this change adds a check for Guaranteed QoS Signed-off-by: ehila <ehila@redhat.com> test: add unit tests for error states Signed-off-by: ehila <ehila@redhat.com>

OpenShift-Rebase-Source: 5ab0f5e

…gated apiservers OpenShift-Rebase-Source: d8adc09

… one UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals This patches the changes from openshift#558 to provide these new functions without changing the behavior for other repos that depend on them, such as library-go. OpenShift-Rebase-Source: 63ed200

… allowed node labels Server side validation of node labels was added in kubernetes#90307. We only disabled kubelet-side validation before to make our node role labels work. UPSTREAM: <carry>: add control plane to allow roles OpenShift-Rebase-Source: 38bfed3 OpenShift-Rebase-Source: aff4434

OpenShift-Rebase-Source: 7bf2f1f

OpenShift-Rebase-Source: 4d74b77

…localhost to force KCM to use localhost set the following flag in kubecontrollermanager (oc edit kubecontrollermanager cluster) unsupportedConfigOverrides: extendedArguments: unsupported-kube-api-over-localhost: - "true" OpenShift-Rebase-Source: 036b11c UPSTREAM: <carry>: allows for switching KCM to talk to Kube API over localhost

OpenShift since 3.x has injected the service serving certificate ca (service ca) bundle into service account token secrets. This was intended to ensure that all pods would be able to easily verify connections to endpoints secured with service serving certificates. Since breaking customer workloads is not an option, and there is no way to ensure that customers are not relying on the service ca bundle being mounted at /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt, it is necessary to continue mounting the service ca bundle in the same location in the bound token projected volumes enabled by the BoundServiceAccountTokenVolume feature (enabled by default in 1.21). A new controller is added to create a configmap per namespace that is annotated for service ca injection. The controller is derived from the controller that creates configmaps for the root ca. The service account admission controller is updated to include a source for the new configmap in the default projected volume definition. UPSTREAM: <carry>: <squash> Add unit testing for service ca configmap publishing This commit should be squashed with: UPSTREAM: <carry>: Ensure service ca is mounted for projected tokens OpenShift-Rebase-Source: d69d054 UPSTREAM: <carry>: Ensure service ca is mounted for projected tokens

… to apiserver_request_total UPSTREAM: <carry>: apiserver: add cluster-policy-controller to system client in apiserver_request_total OpenShift-Rebase-Source: d86823d UPSTREAM: <carry>: apiserver: add system_client=kube-{apiserver,cm,s} to apiserver_request_total Fix TestOpenAPIRequestMetrics unit test.

This is a short term fix, once we improve the cert rotation logic in library-go that does not depend on this hack, then we can remove this carry patch. squash with the previous PR during the rebase openshift#1924 squash with the previous PRs during the rebase openshift#1924 openshift#1929

…iserver OpenShift-Rebase-Source: fb90ed6

…gration test OpenShift-Rebase-Source: 2f4c829 UPSTREAM: 103612: tolerate additional, but congruent, events for integration test

…a.crt for migration compatibility OpenShift-Rebase-Source: bf2b5fa

…ted edit role" OpenShift has an admission controller to prevent restricted Endpoints changes, and there's no reason to block non-restricted changes (such as modifying the annotations of an Endpoints, which is done by "oc idle"). This reverts commit 416efda. OpenShift-Rebase-Source: 239b9ed

…getting false positives until the server becomes ready the availability checks depend on fully initialized SDN OpenShift carries a few reachability checks that affect /readyz protocol we skip posting failures to avoid getting false positives until the server becomes ready UPSTREAM: <carry>: skip posting failures to aggregated APIs to avoid getting false positives until the server becomes ready marks availability of the server before checking the aggregate APIs as it can change as we are running the checks. in that case, skip posting failures to avoid false positives. note on the next rebase please squash with the previous commit UPSTREAM: <carry>: expose HasBeenReady lifecycle signal OpenShift-Rebase-Source: 8558e88

UPSTREAM: <carry>: change opt-in due to upstream revert OpenShift-Rebase-Source: cd08005

OpenShift-Rebase-Source: 3b2555a

… for GC and Namespace controllers In general, setting the header will result in getting 429 when the server hasn't been ready. This prevents certain controllers like GC, Namespace from accidentally removing resources when the caches haven't been fully synchronized. OpenShift-Rebase-Source: 2ebf199

UPSTREAM: <carry>: Force releasing the lock on exit for KS squash with UPSTREAM: <carry>: Release lock on KCM and KS termination OpenShift-Rebase-Source: fc91252 UPSTREAM: <carry>: Release lock on KCM and KS termination

OpenShift-Rebase-Source: 2e5064e

…vice account token should be auto-generated OpenShift-Rebase-Source: a031438 UPSTREAM: <carry>: fix [sig-auth] ServiceAccounts no secret-based service account token should be auto-generated

…eady OpenShift-Rebase-Source: fc3523f

… changes that SCC will eventually make to the pod UPSTREAM: <carry>: pod-security: don't fail on SCC admission error If we propagate SCC admission error during pod extraction to PodSecurity admission, the latter will log the error instead of continuing with unmutated pod spec, and so we will not get a validation error in either the audit logs or as a warning. OpenShift-Rebase-Source: 6fe5c8f OpenShift-Rebase-Source: b4e019f UPSTREAM: <carry>: SCC pod extractor: assume default SA if SA is empty

We need this in order to be able to retrieve better reports from PodSecurityViolation alerts. UPSTREAM: <carry>: PSa metrics: unset ocp_namespace on non-platform namespaces

…aged is enabled Previously, cpu load balancing was enabled in cri-o by manually changing the sched_domain of cpus in sysfs. However, RHEL 9 dropped support for this knob, instead requiring it be changed in cgroups directly. To enable cpu load balancing on cgroupv1, the specified cgroup must have cpuset.sched_load_balance set to 0, as well as all of that cgroup's parents, plus all of the cgroups that contain a subset of the cpus that load balancing is disabled for. By default, all cpusets inherit the set from their parent and sched_load_balance as 1. Since we need to keep the cpus that need load balancing disabled in the root cgroup, all slices will inherit the full cpuset. Rather than rebalancing every cgroup whenever a new guaranteed cpuset cgroup is created, the approach this PR takes is to set load balancing to disabled for all slices. Since slices definitionally don't have any processes in them, setting load balancing won't affect the actual scheduling decisions of the kernel. All it will do is open the opportunity for CRI-O to set the actually set load balancing to disabled for containers that request it. Signed-off-by: Peter Hunt <pehunt@redhat.com> UPSTREAM: <carry>: kubelet/cm: disable cpu load balancing on slices when using static cpu manager policy There are situations where cpu load balance disabling is desired when the kubelet is not in managed state. Instead of using that condition, set the cpu load balancing parameter for new slices when the cpu policy is static Signed-off-by: Peter Hunt <pehunt@redhat.com> UPSTREAM: <carry>: cm: reorder setting of sched_load_balance for sandbox slice If we call mgr.Apply() first, libcontainer's cpusetCopyIfNeeded() will copy the parent cpuset and set load balancing to 1 by default. This causes the kernel to set the cpus to not load balanced for a brief moment which causes churn. instead, create the cgroup and set load balance, then have Apply() copy the values into it. Signed-off-by: Peter Hunt <pehunt@redhat.com> UPSTREAM: <carry>: kubelet/cm: use MkdirAll when creating cpuset to ignore file exists error Signed-off-by: Peter Hunt <pehunt@redhat.com>

If it is useful we will combine this with the following carry: 20caad9: UPSTREAM: 115328: annotate early and late requests UPSTREAM: <carry>: add conditional shutdown response header

…util/managedfields Some of the code we use in openshift-tests was recently made internal in kubernetes#115065. This patch exposes the code we need there.

…rnetes.default.svc, don't wait for aggregated availability

…roups that have kinds that are served by both CRDs and external apiservers (eg openshift-apiserver) this includes: - authorization.openshift.io (rolebindingrestrictions served by a CRD) - security.openshift.io (securitycontextconstraints served by a CRD) - quota.openshift.io (clusterresourcequotas served by a CRD) By merging all sources, we ensure that kinds served by a CRD will have openapi discovery and spec available even when openshift-apiserver is unavailable.

…self-SARs that have user:check-access Otherwise, the request will inherit any scopes that an access token might have and the scopeAuthorizer will deny the access review if the scopes do not include user:full

This commit renews openshift#327 What has changed compared to the original PR is: - The retryClient interface has been adapted to storage.Interface. - The isRetriableEtcdError method has been completely changed; it seems that previously the error we wanted to retry was not being retried. Even the unit tests were failing. Overall, I still think this is not the correct fix. The proper fix should be added to the etcd client. UPSTREAM: <carry>: retry etcd Unavailable errors This is the second commit for the retry logic. This commit adds unit tests and slightly improves the logging. During a rebase squash with the previous one.

When a PerformanceProfile configures a node for cpu partitioning, it also lets OVS use all the cpus available to burstable pods. To be able to do that, OVS was moved to its own slice and that slice needs to be re-added to cAdvisor for monitoring purposes.

Signed-off-by: Harshal Patil <harpatil@redhat.com>

Kubelet should advertise the shared cpus as extedned resources. This has the benefit of limiting the amount of containers that can request an access to the shared cpus. For more information see - openshift/enhancements#1396 Signed-off-by: Talor Itzhak <titzhak@redhat.com>

…force 2nd labeling to make tests work

This commit needs to be carried until we rebase onto Kube 1.31. We have backported the library changes to 1.28, which means they can then be used in 1.29. Upstream, they were only introduced in 1.30 which means they wouldn't be usable until 1.31. This allows us to improve our API validation from OpenShift 4.16 onwards, instead of OpenShift 4.18 onwards. UPSTREAM: <carry>: Set up CEL IP/CIDR library from 4.14 onwards Carry until K8s 1.31 rebase.

Adding a new mutation plugin that handles the following: 1. In case of `workload.openshift.io/enable-shared-cpus` request, it adds an annotation to hint runtime about the request. runtime is not aware of extended resources, hence we need the annotation. 2. It validates the pod's QoS class and return an error if it's not a guaranteed QoS class 3. It validates that no more than a single resource is being request. 4. It validates that the pod deployed in a namespace that has mixedcpus workloads allowed annotation. For more information see - openshift/enhancements#1396 Signed-off-by: Talor Itzhak <titzhak@redhat.com> UPSTREAM: <carry>: Update management webhook pod admission logic Updating the logic for pod admission to allow a pod creation with workload partitioning annotations to be run in a namespace that has no workload allow annoations. The pod will be stripped of its workload annotations and treated as if it were normal, a warning annoation will be placed to note the behavior on the pod. Signed-off-by: ehila <ehila@redhat.com> UPSTREAM: <carry>: add support for cpu limits into management workloads Added support to allow workload partitioning to use the CPU limits for a container, to allow the runtime to make better decisions around workload cpu quotas we are passing down the cpu limit as part of the cpulimit value in the annotation. CRI-O will take that information and calculate the quota per node. This should support situations where workloads might have different cpu period overrides assigned. Updated kubelet for static pods and the admission webhook for regular to support cpu limits. Updated unit test to reflect changes. Signed-off-by: ehila <ehila@redhat.com>

…ject openshift feature gates into pkg/features Signed-off-by: Swarup Ghosh <swghosh@redhat.com>

…olumn The logic is not exressible via JSONPath. Hence, if we want this, we have to help a little with this custom column writer. OpenShift-Rebase-Source: 633a422

- we tie the shutdown events with the UID of the first (shutdown initiated), this provides us with a more deterministic way to compute shutdown duration from these events - move code snippets from the upstream file to openshift specific patch file, it reduces chance of code conflict Note for rebase: squash it into the following commit cfbb6d6 UPSTREAM: <carry>: create termination events

PersistentVolumeLabel admission plugin is being disabled in OCP 4.16. Skip all tests that require it.

…provided-node-ip annotation

The node.status.addresses logic grew organically and with weird semantics, this commit try to document existing semantics when the kubelet uses an external cloud provider and recover the same behavior existing pre-1.29. The node.status.addresses can be populated by the kubelet at startup or delegated to the external cloud provider. If the --node-ip flag is set to an IP in the node, the kubelet will add an annotation to the Node object that will be respected by the external cloud providers, no new IP addresses will be added for the same address type. If the IP set in the --node-ip flag is `0.0.0.0` or `::`, the kubelet will initialize the node with the default address of the corresponding IP family of the unspecified address, and the cloud-provider will override it later.

Remove PersistentVolumeLabel admission plugin from the default list of enabled admissions. The plugin is deprecated upstream and most of its functionality was removed in 1.29 and 1.30. This is continuation of openshift/cluster-kube-apiserver-operator#1693, the admission is enabled on multiple places.

…- revert This reverts commit 79b249c.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] k8s 1.31 #2004

[WIP] k8s 1.31 #2004

Commits on Jun 26, 2024

Commits on Jun 27, 2024

Commits on Jun 28, 2024

Commits on Jun 29, 2024

Commits on Jun 30, 2024

Commits on Jul 1, 2024