MON-3802: implement cross-namespace rules for UWM #2307

simonpasquier · 2024-04-09T14:56:15Z

This change introduces a way to deploy user-defined rules which are not
scoped to their namespace of origin.

To enable the feature, a user-defined monitoring admin needs to
configure at least one namespace in the UWM ConfigMap:

    kind: ConfigMap
    apiVersion: v1
    metadata:
      name: user-workload-monitoring-config
      namespace: openshift-user-workload-monitoring
    data:
      config.yaml: |-
        namespacesWithoutLabelEnforcement: [ user-monitoring-shared ]

For all PrometheusRule objects defined in the user-monitoring-shared
namespace, Prometheus and Thanos Ruler evaluate the PromQL expressions
without enforcing the namespace label of origin. It makes it possible to
have generic rules that get applied to all (or a subset of) the user
projects instead of having individual rule objects in each user project.

The capability is enabled by default but a cluster admin can decide to
disable it in the CMO ConfigMap:

    kind: ConfigMap
    apiVersion: v1
    metadata:
      name: cluster-monitoring-config
      namespace: openshift-monitoring
    data:
      config.yaml: |-
        userWorkloadEnabled: true
        userWorkload:
         rulesWithoutLabelEnforcementAllowed: false

For example, a user-defined admin can create a single rule that fires
when a user namespace doesn't enforce the Restricted pod security
policy.

    apiVersion: monitoring.coreos.com/v1
    kind: PrometheusRule
    metadata:
      name: security
      namespace: user-monitoring-shared
    spec:
      groups:
        - name: pod-security-policy
          rules:
            - alert: "NamespaceNotEnforcingRestrictedPolicy"
              expr: kube_namespace_labels{namespace!~"(openshift|kube).*|default",label_pod_security_kubernetes_io_enforce!="restricted"}
              for: 5m
              annotations:
                summary: "Restricted policy not enforced"
                description: "Namespace {{ $labels.namespace }} doesn't enforce the Restricted pod security policy."
              labels:
                severity: warning

I added CHANGELOG entry for this change.
No user facing changes, so no entry in CHANGELOG was needed.

openshift-ci-robot · 2024-04-09T14:56:19Z

@simonpasquier: This pull request references MON-3802 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.16.0" version, but no target version was set.

In response to this:

I added CHANGELOG entry for this change.

No user facing changes, so no entry in CHANGELOG was needed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2024-04-09T14:56:39Z

@simonpasquier: This pull request references MON-3802 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.16.0" version, but no target version was set.

In response to this:

I added CHANGELOG entry for this change.

No user facing changes, so no entry in CHANGELOG was needed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2024-04-09T14:58:49Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: simonpasquier

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [simonpasquier]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

simonpasquier · 2024-04-12T07:46:49Z

/skip

openshift-ci-robot · 2024-04-12T09:24:52Z

@simonpasquier: This pull request references MON-3802 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.16.0" version, but no target version was set.

In response to this:

This change introduces a way to deploy user-defined rules which are not
scoped to their namespace of origin.

To enable the feature, a user-defined monitoring admin needs to
configure at least one namespace in the UWM ConfigMap:
   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |-
       namespacesWithoutLabelEnforcement: [ user-monitoring-shared ]
For all PrometheusRule objects defined in the user-monitoring-shared
namespace, Prometheus and Thanos Ruler evaluate the PromQL expressions
without enforcing the namespace label of origin. It makes it possible to
have generic rules that get applied to all (or a subset of) the user
projects instead of having individual rule objects in each user project.

The capability is enabled by default but a cluster admin can decide to
disable it in the CMO ConfigMap:
   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |-
       userWorkloadEnabled: true
       userWorkload:
        rulesWithoutNamespaceLabelEnforcementEnabled: false
I added CHANGELOG entry for this change.

No user facing changes, so no entry in CHANGELOG was needed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

simonpasquier · 2024-04-12T09:25:48Z

/cc @bburt-rh
/cc @jan--f

openshift-ci-robot · 2024-04-12T09:43:16Z

@simonpasquier: This pull request references MON-3802 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.16.0" version, but no target version was set.

In response to this:

This change introduces a way to deploy user-defined rules which are not
scoped to their namespace of origin.

To enable the feature, a user-defined monitoring admin needs to
configure at least one namespace in the UWM ConfigMap:
   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |-
       namespacesWithoutLabelEnforcement: [ user-monitoring-shared ]
For all PrometheusRule objects defined in the user-monitoring-shared
namespace, Prometheus and Thanos Ruler evaluate the PromQL expressions
without enforcing the namespace label of origin. It makes it possible to
have generic rules that get applied to all (or a subset of) the user
projects instead of having individual rule objects in each user project.

The capability is enabled by default but a cluster admin can decide to
disable it in the CMO ConfigMap:
   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |-
       userWorkloadEnabled: true
       userWorkload:
        rulesWithoutLabelEnforcementAllowed: false
I added CHANGELOG entry for this change.

No user facing changes, so no entry in CHANGELOG was needed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2024-04-12T09:58:15Z

@simonpasquier: This pull request references MON-3802 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.16.0" version, but no target version was set.

In response to this:

This change introduces a way to deploy user-defined rules which are not
scoped to their namespace of origin.

To enable the feature, a user-defined monitoring admin needs to
configure at least one namespace in the UWM ConfigMap:
   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |-
       namespacesWithoutLabelEnforcement: [ user-monitoring-shared ]
For all PrometheusRule objects defined in the user-monitoring-shared
namespace, Prometheus and Thanos Ruler evaluate the PromQL expressions
without enforcing the namespace label of origin. It makes it possible to
have generic rules that get applied to all (or a subset of) the user
projects instead of having individual rule objects in each user project.

The capability is enabled by default but a cluster admin can decide to
disable it in the CMO ConfigMap:
   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |-
       userWorkloadEnabled: true
       userWorkload:
        rulesWithoutLabelEnforcementAllowed: false
For example, a user-defined admin can create a single rule that fires
when a user namespace doesn't enforce the Restricted pod security
policy.
   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: security
     namespace: user-monitoring-shared
   spec:
     groups:
       - name: pod-security-policy
         rules:
           - alert: "NamespaceNotEnforcingRestrictedPolicy"
             expr: kube_namespace_labels{namespace!~"(openshift|kube).*|default)",label_pod_security_kubernetes_io_enforce!="restricted"}
             for: 5m
             annotations:
               summary: "Restricted policy not enforced"
               description: "Namespace {{ $labels.namespace }} doesn't enforce the Restricted pod security policy."
             labels:
               severity: warning
I added CHANGELOG entry for this change.

No user facing changes, so no entry in CHANGELOG was needed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

simonpasquier · 2024-04-15T09:17:36Z

current status:

Need to fix the e2e tests when cross-namespace rules are disabled.
I found an issue with the dev console: the alerts generated from cross-namespace rules aren't visible because the console queries for the rules with the exact namespace value.

jan--f · 2024-04-15T11:45:31Z

As mentioned in the sync earlier, I would like to make sure we make it very clear to the user what we mean by cross-namespace. Iiuc this means user namespace as well as system namespaces. Since this currently talks about cross namespace in the context of UWM, I think we should make it explicit that system namespaces can be queried as well for such rules (if that is actually the case).
I'm sure @bburt-rh would know best how to phrase that.

simonpasquier · 2024-04-16T12:28:25Z

summarizing offline discussions about the console issue

The problem is that cross-namespace rules won't be displayed in the developer console.

As an example, I've deployed a cross-namespace rule (NamespaceNotEnforcingRestrictedPolicy ) into the user-monitoring-shared namespace. The rule fires an alert for the ns1 namespace but the alert isn't visible in the dev console:

It is visible in the admin console though:

In terms of user experience, it is less than ideal since a user with only access to the ns1 project can't see the alert being active and they can't silence it if they receive an Alertmanager notification for it.

The reason behind the issue is that the console uses the /api/v1/rules endpoint exposed by prom-label-proxy which will only return alerting rules with a static namespace="<selected namespace>" label.

Possible options being discussed:

Do nothing in the short term and accept it as a know issue.
Have the dev console use the /api/v1/alerts endpoint instead. The problem is that we won't have access to the alert definition (including the PromQL expression) which makes it hard for the user to understand the cause and investigate further.
Modify prom-label-proxy to return any rule that matches the given namespace or that has an alert matching the given namespace. It looks like the most appropriate solution and something that also makes outside of OCP.

simonpasquier · 2024-04-16T15:13:39Z

/hold

simonpasquier · 2024-04-17T12:53:15Z

Modify prom-label-proxy to return any rule that matches the given namespace or that has an alert matching the given namespace. It looks like the most appropriate solution and something that also makes outside of OCP.

I tested this with openshift/prom-label-proxy#369 and it's almost working. When clicking on the alert link to open the PromQL expression in the metrics dashboard, prom-label-proxy replies with a 400 status code and label matcher value (namespace="user-monitoring-shared") conflicts with injected value (namespace!~"(openshift|kube).*|default"). This is because prom-label-proxy runs with -error-on-replace.

openshift-ci-robot · 2024-06-17T10:27:56Z

@simonpasquier: This pull request references MON-3802 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.17.0" version, but no target version was set.

In response to this:

This change introduces a way to deploy user-defined rules which are not
scoped to their namespace of origin.

To enable the feature, a user-defined monitoring admin needs to
configure at least one namespace in the UWM ConfigMap:
   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |-
       namespacesWithoutLabelEnforcement: [ user-monitoring-shared ]
For all PrometheusRule objects defined in the user-monitoring-shared
namespace, Prometheus and Thanos Ruler evaluate the PromQL expressions
without enforcing the namespace label of origin. It makes it possible to
have generic rules that get applied to all (or a subset of) the user
projects instead of having individual rule objects in each user project.

The capability is enabled by default but a cluster admin can decide to
disable it in the CMO ConfigMap:
   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |-
       userWorkloadEnabled: true
       userWorkload:
        rulesWithoutLabelEnforcementAllowed: false
For example, a user-defined admin can create a single rule that fires
when a user namespace doesn't enforce the Restricted pod security
policy.
   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: security
     namespace: user-monitoring-shared
   spec:
     groups:
       - name: pod-security-policy
         rules:
           - alert: "NamespaceNotEnforcingRestrictedPolicy"
             expr: kube_namespace_labels{namespace!~"(openshift|kube).*|default",label_pod_security_kubernetes_io_enforce!="restricted"}
             for: 5m
             annotations:
               summary: "Restricted policy not enforced"
               description: "Namespace {{ $labels.namespace }} doesn't enforce the Restricted pod security policy."
             labels:
               severity: warning
I added CHANGELOG entry for this change.

No user facing changes, so no entry in CHANGELOG was needed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

simonpasquier · 2024-08-08T07:20:52Z

/hold cancel

simonpasquier · 2024-08-08T07:30:00Z

/retest-required

simonpasquier · 2024-08-08T10:47:31Z

/retest-required

simonpasquier · 2024-08-08T12:20:43Z

/skip

This change introduces a way to deploy user-defined rules which are not scoped to their namespace of origin. To enable the feature, a user-defined monitoring admin needs to configure at least one namespace in the UWM ConfigMap: ``` kind: ConfigMap apiVersion: v1 metadata: name: user-workload-monitoring-config namespace: openshift-user-workload-monitoring data: config.yaml: |- namespacesWithoutLabelEnforcement: [ user-monitoring-shared ] ``` For all `PrometheusRule` objects defined in the `user-monitoring-shared` namespace, Prometheus and Thanos Ruler evaluate the PromQL expressions without enforcing the namespace label of origin. It makes it possible to have generic rules that get applied to all (or a subset of) the user projects instead of having individual rule objects in each user project. The rules can also query for platform namespaces only. The capability is enabled by default but a cluster admin can decide to disable it in the CMO ConfigMap: ``` kind: ConfigMap apiVersion: v1 metadata: name: cluster-monitoring-config namespace: openshift-monitoring data: config.yaml: |- userWorkloadEnabled: true userWorkload: rulesWithoutLabelEnforcementAllowed: false ``` For example, a user-defined monitoring admin can create a single rule that fires when a user namespace doesn't enforce the Restricted pod security policy: ``` apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: security namespace: user-monitoring-shared spec: groups: - name: pod-security-policy rules: - alert: "NamespaceNotEnforcingRestrictedPolicy" expr: kube_namespace_labels{namespace!~"(openshift|kube).*|default",label_pod_security_kubernetes_io_enforce!="restricted"} for: 5m annotations: summary: "Restricted policy not enforced" description: "Namespace {{ $labels.namespace }} doesn't enforce the Restricted pod security policy." labels: severity: warning ``` Signed-off-by: Simon Pasquier <spasquie@redhat.com>

openshift-ci · 2024-08-08T15:19:09Z

@simonpasquier: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

simonpasquier · 2024-08-08T15:31:35Z

/assign @machine424

Tai-RedHat · 2024-08-13T04:53:08Z

PR tested with cluster-bot, other user-defined namespace could trigger the alert in user-monitoring-shared namespace.
test case: https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-75384
/label qe-approved

openshift-ci-robot · 2024-08-13T04:53:13Z

@simonpasquier: This pull request references MON-3802 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.18.0" version, but no target version was set.

In response to this:

This change introduces a way to deploy user-defined rules which are not
scoped to their namespace of origin.

To enable the feature, a user-defined monitoring admin needs to
configure at least one namespace in the UWM ConfigMap:
   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |-
       namespacesWithoutLabelEnforcement: [ user-monitoring-shared ]
For all PrometheusRule objects defined in the user-monitoring-shared
namespace, Prometheus and Thanos Ruler evaluate the PromQL expressions
without enforcing the namespace label of origin. It makes it possible to
have generic rules that get applied to all (or a subset of) the user
projects instead of having individual rule objects in each user project.

The capability is enabled by default but a cluster admin can decide to
disable it in the CMO ConfigMap:
   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |-
       userWorkloadEnabled: true
       userWorkload:
        rulesWithoutLabelEnforcementAllowed: false
For example, a user-defined admin can create a single rule that fires
when a user namespace doesn't enforce the Restricted pod security
policy.
   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: security
     namespace: user-monitoring-shared
   spec:
     groups:
       - name: pod-security-policy
         rules:
           - alert: "NamespaceNotEnforcingRestrictedPolicy"
             expr: kube_namespace_labels{namespace!~"(openshift|kube).*|default",label_pod_security_kubernetes_io_enforce!="restricted"}
             for: 5m
             annotations:
               summary: "Restricted policy not enforced"
               description: "Namespace {{ $labels.namespace }} doesn't enforce the Restricted pod security policy."
             labels:
               severity: warning
I added CHANGELOG entry for this change.

No user facing changes, so no entry in CHANGELOG was needed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Tai-RedHat · 2024-08-13T05:03:53Z

If I set two or even more user-monitoring-shared namespaces, and then create same prometheusrules to each namespaces. This will result in repeated alerts. Should we remind or restrict users in this situation?
config.yaml: 'namespacesWithoutLabelEnforcement: [ns1, ns2]'

simonpasquier · 2024-08-14T14:07:03Z

If I set two or even more user-monitoring-shared namespaces, and then create same prometheusrules to each namespaces. This will result in repeated alerts. Should we remind or restrict users in this situation?

Thanks for the testing @Tai-RedHat! I don't see a strong reason to prevent this situation (and it would be quite hard to detect). But it would be good to mention in the docs.

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Apr 9, 2024

simonpasquier changed the title ~~MON-3802: implement cross-namespace rules for UWM~~ [WIP] MON-3802: implement cross-namespace rules for UWM Apr 9, 2024

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 9, 2024

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 9, 2024

simonpasquier force-pushed the MON-3802 branch 5 times, most recently from 5db1763 to 3b2e644 Compare April 11, 2024 13:38

simonpasquier changed the title ~~[WIP] MON-3802: implement cross-namespace rules for UWM~~ MON-3802: implement cross-namespace rules for UWM Apr 12, 2024

simonpasquier force-pushed the MON-3802 branch from 3b2e644 to 6949f31 Compare April 12, 2024 09:23

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 12, 2024

openshift-ci bot requested review from bburt-rh and jan--f April 12, 2024 09:25

simonpasquier force-pushed the MON-3802 branch from 6949f31 to 03006be Compare April 12, 2024 09:42

simonpasquier force-pushed the MON-3802 branch 2 times, most recently from d319656 to 22bb813 Compare April 15, 2024 08:52

simonpasquier force-pushed the MON-3802 branch 2 times, most recently from f9d4f45 to 5b874d1 Compare April 16, 2024 06:25

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 16, 2024

simonpasquier force-pushed the MON-3802 branch from 5b874d1 to 409d96a Compare April 16, 2024 07:00

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 16, 2024

simonpasquier force-pushed the MON-3802 branch from 409d96a to 591fe80 Compare April 16, 2024 12:13

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 16, 2024

simonpasquier force-pushed the MON-3802 branch from 591fe80 to 509f8a8 Compare June 17, 2024 08:59

simonpasquier force-pushed the MON-3802 branch from 509f8a8 to 6a77b65 Compare July 31, 2024 10:17

simonpasquier force-pushed the MON-3802 branch 2 times, most recently from 1051350 to 77b2f1b Compare August 8, 2024 07:14

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 8, 2024

simonpasquier added 2 commits August 8, 2024 14:56

CONTRIBUTING.md: clean-up

4f7f29a

simonpasquier force-pushed the MON-3802 branch from 77b2f1b to 942b85e Compare August 8, 2024 13:00

openshift-ci bot assigned machine424 Aug 8, 2024

openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label Aug 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MON-3802: implement cross-namespace rules for UWM #2307

MON-3802: implement cross-namespace rules for UWM #2307

simonpasquier commented Apr 9, 2024 •

edited

Loading

openshift-ci-robot commented Apr 9, 2024 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Apr 9, 2024 •

edited by openshift-ci bot

Loading

openshift-ci bot commented Apr 9, 2024

simonpasquier commented Apr 12, 2024

openshift-ci-robot commented Apr 12, 2024 •

edited by openshift-ci bot

Loading

simonpasquier commented Apr 12, 2024

openshift-ci-robot commented Apr 12, 2024 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Apr 12, 2024 •

edited by openshift-ci bot

Loading

simonpasquier commented Apr 15, 2024

jan--f commented Apr 15, 2024

simonpasquier commented Apr 16, 2024

simonpasquier commented Apr 16, 2024

simonpasquier commented Apr 17, 2024

openshift-ci-robot commented Jun 17, 2024 •

edited by openshift-ci bot

Loading

simonpasquier commented Aug 8, 2024

simonpasquier commented Aug 8, 2024

simonpasquier commented Aug 8, 2024

simonpasquier commented Aug 8, 2024

openshift-ci bot commented Aug 8, 2024

simonpasquier commented Aug 8, 2024

Tai-RedHat commented Aug 13, 2024

openshift-ci-robot commented Aug 13, 2024 •

edited by openshift-ci bot

Loading

Tai-RedHat commented Aug 13, 2024

simonpasquier commented Aug 14, 2024

MON-3802: implement cross-namespace rules for UWM #2307

Are you sure you want to change the base?

MON-3802: implement cross-namespace rules for UWM #2307

Conversation

simonpasquier commented Apr 9, 2024 • edited Loading

openshift-ci-robot commented Apr 9, 2024 • edited by openshift-ci bot Loading

openshift-ci-robot commented Apr 9, 2024 • edited by openshift-ci bot Loading

openshift-ci bot commented Apr 9, 2024

simonpasquier commented Apr 12, 2024

openshift-ci-robot commented Apr 12, 2024 • edited by openshift-ci bot Loading

simonpasquier commented Apr 12, 2024

openshift-ci-robot commented Apr 12, 2024 • edited by openshift-ci bot Loading

openshift-ci-robot commented Apr 12, 2024 • edited by openshift-ci bot Loading

simonpasquier commented Apr 15, 2024

jan--f commented Apr 15, 2024

simonpasquier commented Apr 16, 2024

simonpasquier commented Apr 16, 2024

simonpasquier commented Apr 17, 2024

openshift-ci-robot commented Jun 17, 2024 • edited by openshift-ci bot Loading

simonpasquier commented Aug 8, 2024

simonpasquier commented Aug 8, 2024

simonpasquier commented Aug 8, 2024

simonpasquier commented Aug 8, 2024

openshift-ci bot commented Aug 8, 2024

simonpasquier commented Aug 8, 2024

Tai-RedHat commented Aug 13, 2024

openshift-ci-robot commented Aug 13, 2024 • edited by openshift-ci bot Loading

Tai-RedHat commented Aug 13, 2024

simonpasquier commented Aug 14, 2024

simonpasquier commented Apr 9, 2024 •

edited

Loading

openshift-ci-robot commented Apr 9, 2024 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Apr 9, 2024 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Apr 12, 2024 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Apr 12, 2024 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Apr 12, 2024 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Jun 17, 2024 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Aug 13, 2024 •

edited by openshift-ci bot

Loading