Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[hcp|kv|disconnected] Various fixes for hcp|kv|disconnected periodics #54062

Merged
merged 2 commits into from
Jul 16, 2024
Merged

[hcp|kv|disconnected] Various fixes for hcp|kv|disconnected periodics #54062

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f7eb2bc
[hcp|kv|disconnected] Various fixes for hcp|kv|disconnected periodics
tiraboschi Jul 5, 2024
7536571
Use profile equinix-ocp-hcp
tiraboschi Jul 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 3 additions & 1 deletion ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.14__periodics.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -220,8 +220,10 @@ tests:
- as: e2e-kubevirt-metal-mce-disconnected
cron: 0 8 * * *
steps:
cluster_profile: equinix-ocp-metal
cluster_profile: equinix-ocp-hcp
env:
CCS_OPERATOR_CHANNELS: stable-4.14,stable-4.14,stable-4.14,stable-4.14
CCS_OPERATOR_PACKAGES: odf-operator,ocs-operator,mcg-operator,odf-csi-addons-operator
HYPERSHIFT_NODE_COUNT: "2"
LVM_OPERATOR_SUB_CHANNEL: stable-4.14
MCE_VERSION: "2.4"
Expand Down
4 changes: 3 additions & 1 deletion ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.15__periodics.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,8 +192,10 @@ tests:
- as: e2e-kubevirt-metal-ovn-mce-disconnected
cron: 0 4 * * *
steps:
cluster_profile: equinix-ocp-metal
cluster_profile: equinix-ocp-hcp
env:
CCS_OPERATOR_CHANNELS: stable-4.15,stable-4.15,stable-4.15,stable-4.15
CCS_OPERATOR_PACKAGES: odf-operator,ocs-operator,mcg-operator,odf-csi-addons-operator
HYPERSHIFT_NODE_COUNT: "2"
LVM_OPERATOR_SUB_CHANNEL: stable-4.15
MCE_VERSION: "2.5"
Expand Down
9 changes: 5 additions & 4 deletions ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.16__periodics.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,18 +186,19 @@ tests:
MCE_VERSION: "2.6"
ODF_OPERATOR_SUB_CHANNEL: stable-4.15
workflow: hypershift-mce-kubevirt-baremetalds-conformance
- as: e2e-kubevirt-metal-ovn-disconnected
- as: e2e-kubevirt-metal-ovn-mce-disconnected
cron: 0 8 * * *
steps:
cluster_profile: equinix-ocp-metal
cluster_profile: equinix-ocp-hcp
env:
CCS_OPERATOR_CHANNELS: stable-4.15,stable-4.15,stable-4.15,stable-4.15
CCS_OPERATOR_PACKAGES: odf-operator,ocs-operator,mcg-operator,odf-csi-addons-operator
HYPERSHIFT_NODE_COUNT: "2"
LVM_OPERATOR_SUB_CHANNEL: stable-4.15
MIRROR_OLM_REMOTE_INDEX: registry.redhat.io/redhat/redhat-operator-index:v4.15
MCE_VERSION: "2.6"
MIRROR_OLM_REMOTE_INDEX: registry.redhat.io/redhat/redhat-operator-index:v4.16
ODF_OPERATOR_SUB_CHANNEL: stable-4.15
workflow: hypershift-kubevirt-baremetalds-disconnected-conformance
workflow: hypershift-mce-kubevirt-baremetalds-disconnected-conformance
- as: e2e-mce-ibmz-ovn-conformance
cron: 0 8 * * *
steps:
Expand Down
2 changes: 1 addition & 1 deletion ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.17__periodics.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,7 @@ tests:
- as: e2e-kubevirt-metal-ovn-disconnected
cron: 0 8 * * *
steps:
cluster_profile: equinix-ocp-metal
cluster_profile: equinix-ocp-hcp
env:
CCS_OPERATOR_CHANNELS: stable-4.15,stable-4.15,stable-4.15,stable-4.15
CCS_OPERATOR_PACKAGES: odf-operator,ocs-operator,mcg-operator,odf-csi-addons-operator
Expand Down
4 changes: 2 additions & 2 deletions ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.14-periodics.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -792,7 +792,7 @@ periodics:
repo: hypershift
labels:
ci-operator.openshift.io/cloud: equinix-ocp-metal
ci-operator.openshift.io/cloud-cluster-profile: equinix-ocp-metal
ci-operator.openshift.io/cloud-cluster-profile: equinix-ocp-hcp
ci-operator.openshift.io/variant: periodics
ci.openshift.io/generator: prowgen
job-release: "4.14"
Expand Down Expand Up @@ -858,7 +858,7 @@ periodics:
secretName: ci-pull-credentials
- name: cluster-profile
secret:
secretName: cluster-secrets-equinix-ocp-metal
secretName: cluster-secrets-equinix-ocp-hcp
- name: manifest-tool-local-pusher
secret:
secretName: manifest-tool-local-pusher
Expand Down
4 changes: 2 additions & 2 deletions ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.15-periodics.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -799,7 +799,7 @@ periodics:
repo: hypershift
labels:
ci-operator.openshift.io/cloud: equinix-ocp-metal
ci-operator.openshift.io/cloud-cluster-profile: equinix-ocp-metal
ci-operator.openshift.io/cloud-cluster-profile: equinix-ocp-hcp
ci-operator.openshift.io/variant: periodics
ci.openshift.io/generator: prowgen
job-release: "4.15"
Expand Down Expand Up @@ -865,7 +865,7 @@ periodics:
secretName: ci-pull-credentials
- name: cluster-profile
secret:
secretName: cluster-secrets-equinix-ocp-metal
secretName: cluster-secrets-equinix-ocp-hcp
- name: manifest-tool-local-pusher
secret:
secretName: manifest-tool-local-pusher
Expand Down
19 changes: 13 additions & 6 deletions ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.16-periodics.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -866,12 +866,19 @@ periodics:
repo: hypershift
labels:
ci-operator.openshift.io/cloud: equinix-ocp-metal
ci-operator.openshift.io/cloud-cluster-profile: equinix-ocp-metal
ci-operator.openshift.io/cloud-cluster-profile: equinix-ocp-hcp
ci-operator.openshift.io/variant: periodics
ci.openshift.io/generator: prowgen
job-release: "4.16"
pj-rehearse.openshift.io/can-be-rehearsed: "true"
name: periodic-ci-openshift-hypershift-release-4.16-periodics-e2e-kubevirt-metal-ovn-disconnected
name: periodic-ci-openshift-hypershift-release-4.16-periodics-e2e-kubevirt-metal-ovn-mce-disconnected
reporter_config:
slack:
channel: '#hypershift-kubevirt-platform'
job_states_to_report:
- failure
report_template: ':volcano: Job *{{.Spec.Job}}* ended with *{{.Status.State}}*.
<{{.Status.URL}}|View logs> :volcano:'
spec:
containers:
- args:
Expand All @@ -880,8 +887,8 @@ periodics:
- --lease-server-credentials-file=/etc/boskos/credentials
- --report-credentials-file=/etc/report/credentials
- --secret-dir=/secrets/ci-pull-credentials
- --secret-dir=/usr/local/e2e-kubevirt-metal-ovn-disconnected-cluster-profile
- --target=e2e-kubevirt-metal-ovn-disconnected
- --secret-dir=/usr/local/e2e-kubevirt-metal-ovn-mce-disconnected-cluster-profile
- --target=e2e-kubevirt-metal-ovn-mce-disconnected
- --variant=periodics
command:
- ci-operator
Expand All @@ -898,7 +905,7 @@ periodics:
- mountPath: /secrets/ci-pull-credentials
name: ci-pull-credentials
readOnly: true
- mountPath: /usr/local/e2e-kubevirt-metal-ovn-disconnected-cluster-profile
- mountPath: /usr/local/e2e-kubevirt-metal-ovn-mce-disconnected-cluster-profile
name: cluster-profile
- mountPath: /secrets/gcs
name: gcs-credentials
Expand All @@ -925,7 +932,7 @@ periodics:
secretName: ci-pull-credentials
- name: cluster-profile
secret:
secretName: cluster-secrets-equinix-ocp-metal
secretName: cluster-secrets-equinix-ocp-hcp
- name: manifest-tool-local-pusher
secret:
secretName: manifest-tool-local-pusher
Expand Down
11 changes: 9 additions & 2 deletions ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.17-periodics.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1109,12 +1109,19 @@ periodics:
repo: hypershift
labels:
ci-operator.openshift.io/cloud: equinix-ocp-metal
ci-operator.openshift.io/cloud-cluster-profile: equinix-ocp-metal
ci-operator.openshift.io/cloud-cluster-profile: equinix-ocp-hcp
ci-operator.openshift.io/variant: periodics
ci.openshift.io/generator: prowgen
job-release: "4.17"
pj-rehearse.openshift.io/can-be-rehearsed: "true"
name: periodic-ci-openshift-hypershift-release-4.17-periodics-e2e-kubevirt-metal-ovn-disconnected
reporter_config:
slack:
channel: '#hypershift-kubevirt-platform'
job_states_to_report:
- failure
report_template: ':volcano: Job *{{.Spec.Job}}* ended with *{{.Status.State}}*.
<{{.Status.URL}}|View logs> :volcano:'
spec:
containers:
- args:
Expand Down Expand Up @@ -1168,7 +1175,7 @@ periodics:
secretName: ci-pull-credentials
- name: cluster-profile
secret:
secretName: cluster-secrets-equinix-ocp-metal
secretName: cluster-secrets-equinix-ocp-hcp
- name: manifest-tool-local-pusher
secret:
secretName: manifest-tool-local-pusher
Expand Down
4 changes: 2 additions & 2 deletions ci-operator/step-registry/hypershift/conformance/hypershift-conformance-chain.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ chain:
FROM=$(oc get -n openshift ${isName} -o=jsonpath={.spec.tags[0].from.name})
EXT_REG=${FROM%%@*}
INT_PULL_URL=${FROM/$EXT_REG/$DEVSCRIPTS_RELEASE_IMAGE_REPO}
oc patch -n openshift ${isName} --type json -p "[{\"op\": \"replace\", \"path\": \"/spec/tags/0/importPolicy/insecure\", \"value\": true}, {\"op\": \"replace\", \"path\": \"/spec/tags/0/from/name\", \"value\": \"${INT_PULL_URL}\"}]"
oc patch -n openshift ${isName} --type json -p "[{\"op\": \"replace\", \"path\": \"/spec/tags/0/importPolicy/insecure\", \"value\": true}, {\"op\": \"replace\", \"path\": \"/spec/tags/0/from\", \"value\": {\"kind\": \"DockerImage\", \"name\": \"${INT_PULL_URL}\"} }]"
done

### tests like "services when running openshift cluster on KubeVirt virtual machines should allow connections to pods from infra cluster pod *"
Expand All @@ -45,7 +45,7 @@ chain:
FROM=$(oc --kubeconfig=${HYPERSHIFT_MANAGEMENT_CLUSTER_KUBECONFIG} get -n openshift ${isName} -o=jsonpath={.spec.tags[0].from.name})
EXT_REG=${FROM%%@*}
INT_PULL_URL=${FROM/$EXT_REG/$DEVSCRIPTS_RELEASE_IMAGE_REPO}
oc patch --kubeconfig=${HYPERSHIFT_MANAGEMENT_CLUSTER_KUBECONFIG} -n openshift ${isName} --type json -p "[{\"op\": \"replace\", \"path\": \"/spec/tags/0/importPolicy/insecure\", \"value\": true}, {\"op\": \"replace\", \"path\": \"/spec/tags/0/from/name\", \"value\": \"${INT_PULL_URL}\"}]"
oc patch --kubeconfig=${HYPERSHIFT_MANAGEMENT_CLUSTER_KUBECONFIG} -n openshift ${isName} --type json -p "[{\"op\": \"replace\", \"path\": \"/spec/tags/0/importPolicy/insecure\", \"value\": true}, {\"op\": \"replace\", \"path\": \"/spec/tags/0/from\", \"value\": {\"kind\": \"DockerImage\", \"name\": \"${INT_PULL_URL}\"} }]"
###
}

Expand Down
1 change: 1 addition & 0 deletions ...nected/conformance/hypershift-kubevirt-baremetalds-disconnected-conformance-workflow.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ workflow:
allow_best_effort_post_steps: true
allow_skip_on_success: true
post:
- ref: hypershift-kubevirt-baremetalds-disconnected-highlightmissingimages
- chain: gather-core-dump
- chain: hypershift-kubevirt-destroy
- chain: baremetalds-ipi-post
Expand Down
19 changes: 19 additions & 0 deletions .../step-registry/hypershift/kubevirt/baremetalds/disconnected/highlightmissingimages/OWNERS
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
approvers:
- csrwng
- enxebre
- sjenning
- imain
- LiangquanLi930
- davidvossel
- orenc1
- tiraboschi
options: {}
reviewers:
- csrwng
- enxebre
- sjenning
- imain
- LiangquanLi930
- davidvossel
- orenc1
- tiraboschi
13 changes: 13 additions & 0 deletions ...ingimages/hypershift-kubevirt-baremetalds-disconnected-highlightmissingimages-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
#!/bin/bash

set -x

echo "--- Management cluster ---"
oc get pods -A -o yaml | grep "Back-off pulling image" | sort | uniq

echo "--- Hosted cluster ---"
if [[ -f "${SHARED_DIR}/nested_kubeconfig" ]]; then
export KUBECONFIG="${SHARED_DIR}/nested_kubeconfig"
oc get pods -A -o yaml | grep "Back-off pulling image" | sort | uniq
fi

25 changes: 25 additions & 0 deletions ...ges/hypershift-kubevirt-baremetalds-disconnected-highlightmissingimages-ref.metadata.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
{
"path": "hypershift/kubevirt/baremetalds/disconnected/highlightmissingimages/hypershift-kubevirt-baremetalds-disconnected-highlightmissingimages-ref.yaml",
"owners": {
"approvers": [
"csrwng",
"enxebre",
"sjenning",
"imain",
"LiangquanLi930",
"davidvossel",
"orenc1",
"tiraboschi"
],
"reviewers": [
"csrwng",
"enxebre",
"sjenning",
"imain",
"LiangquanLi930",
"davidvossel",
"orenc1",
"tiraboschi"
]
}
}
12 changes: 12 additions & 0 deletions ...issingimages/hypershift-kubevirt-baremetalds-disconnected-highlightmissingimages-ref.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
ref:
as: hypershift-kubevirt-baremetalds-disconnected-highlightmissingimages
from: cli
grace_period: 10m
commands: hypershift-kubevirt-baremetalds-disconnected-highlightmissingimages-commands.sh
resources:
requests:
cpu: 1000m
memory: 500Mi
timeout: 10m0s
documentation: |-
This step to quickly identify missing images on disconnected test lanes
31 changes: 31 additions & 0 deletions ...baremetalds/redhat-operators/hypershift-kubevirt-baremetalds-redhat-operators-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,37 @@ END

echo "7: Waiting for the new ImageContentSourcePolicy to be updated on machines"
oc wait clusteroperators/machine-config --for=condition=Upgradeable=true --timeout=15m

# TODO: why do we need this? do we have a bug?
echo "8: explicitly fix IDMS for CSI sidecar images"

oc apply -f - <<END
apiVersion: config.openshift.io/v1
kind: ImageDigestMirrorSet
metadata:
name: cs-ccs-local-fixes
spec:
imageDigestMirrors:
- mirrors:
- virthost.ostest.test.metalkube.org:5000/openshift4/ose-csi-external-provisioner
source: registry.redhat.io/openshift4/ose-csi-external-provisioner
- mirrors:
- virthost.ostest.test.metalkube.org:5000/openshift4/ose-csi-external-resizer
source: registry.redhat.io/openshift4/ose-csi-external-resizer
- mirrors:
- virthost.ostest.test.metalkube.org:5000/openshift4/ose-csi-external-snapshotter-rhel9
source: registry.redhat.io/openshift4/ose-csi-external-snapshotter-rhel9
- mirrors:
- virthost.ostest.test.metalkube.org:5000/openshift4/ose-csi-external-snapshotter-rhel8
source: registry.redhat.io/openshift4/ose-csi-external-snapshotter-rhel8
- mirrors:
- virthost.ostest.test.metalkube.org:5000/openshift4/ose-csi-node-driver-registrar
source: registry.redhat.io/openshift4/ose-csi-node-driver-registrar
- mirrors:
- virthost.ostest.test.metalkube.org:5000/openshift4/ose-kube-rbac-proxy
source: registry.redhat.io/openshift4/ose-kube-rbac-proxy
END

EOF
}

Expand Down
1 change: 1 addition & 0 deletions ci-operator/step-registry/hypershift/kubevirt/create/hypershift-kubevirt-create-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,7 @@ then

fi

oc create namespace "${CLUSTER_NAMESPACE_PREFIX}" --dry-run=client -o yaml | oc apply -f -
oc create ns "${CLUSTER_NAMESPACE_PREFIX}-${CLUSTER_NAME}"
if [[ -n "${ATTACH_DEFAULT_NETWORK}" ]]; then
oc apply -f - <<EOF
Expand Down
1 change: 1 addition & 0 deletions ...ed/conformance/hypershift-mce-kubevirt-baremetalds-disconnected-conformance-workflow.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ workflow:
allow_best_effort_post_steps: true
allow_skip_on_success: true
post:
- ref: hypershift-kubevirt-baremetalds-disconnected-highlightmissingimages
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAIK, hypershift-mce-dump can accomplish the same thing.
E.g: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_release/54062/rehearse-54062-periodic-ci-openshift-hypershift-release-4.14-periodics-e2e-kubevirt-metal-mce-disconnected/1812774648182476800/artifacts/e2e-kubevirt-metal-mce-disconnected/hypershift-mce-dump/artifacts/hostedcluster-6d047c994a7bcc9e751e/

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm also adding it to hypershift-kubevirt-baremetalds-disconnected-conformance-workflow.yaml which is not calling hypershift-mce-dump, but more than that is a matter of visibility: for sure we already have all the info we need to debug an issue in the must-gather archives we are already collecting, but getting at least one of the images not correctly mirrored on the internal registry is by far the most common issue in those disconnected lanes and this new step will highlight it if the issue is really there.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay, Thanks

- ref: hypershift-mce-dump
- chain: gather-core-dump
- chain: hypershift-mce-kubevirt-destroy
Expand Down