NETOBSERV-1112: This patch fixes a bug where RTT was not visible for flow logs at times. #159

dushyantbehl · 2023-07-20T10:17:37Z

The calculation of RTT in ebpf agent was initially implemented to be doing something like this,

We were storing the timestamp of outgoing SYN and incoming ACK, calculating the RTT as Incoming ACK - Outgoing SYN.

This works for calculations done on both client and server endpoints as both have one SYN which is sent out and one ack which is received in response to that to start a TCP connection.

ebpf agent on the other hand does not connect only at a single interface in Openshift environment. If we look at the same communication being done from inside a container it would look like this.

In this case the container contains a veth pair and ebpf agent attaches to the veth interface in the root namespace, let's call veth for a vpeer in the container namespace.
Any packet emerging from vpeer which is put at the egress queue of container will come at the ingress queue of the veth interface. Hence if we try to calculate RTT using the previous logic of outgoing SYN and incoming ACK then we would end up calculating the timestamp of SYN(Y) and ACK(Y) which will be just the response time of the container and not a correct value of the RTT.

Also note that the ebpf agent attaches to many interfaces in Openshift and can attach to interfaces like eth0 which always act like a Client or Server endpoints and hence the RTT calculation of Incoming ACK - Outgoing SYN will work for such endpoints.

So to account for both the cases we changed the RTT calculation in a simple way, we simply take the RTT of SYN/SYN+ACK/ACK separately, i.e. we will calculate RTT of SYN(X) and ACK(X) and SYN(Y) and ACK(Y) and simply take the maximum of both values.
This is needed because ebpf agent attaches at both type of interfaces, root veth interfaces which act like middleboxes and eth0 interfaces which act like final connection endpoints.

Note that the flows are in reverse direction so we need to calculate RTT for reverse flows in ebpf agent but its pretty easy to do.
Also, this calculation is interface specific so we need to add interface id to the sequence identifier.

dushyantbehl · 2023-07-20T10:19:39Z

/ok-to-test

github-actions · 2023-07-20T10:21:32Z

New image:
quay.io/netobserv/netobserv-ebpf-agent:3f822c7

It will expire after two weeks.

To deploy this build, run from the operator repo, assuming the operator is running:

USER=netobserv VERSION=3f822c7 make set-agent-image

codecov · 2023-07-20T10:23:02Z

Codecov Report

Merging #159 (e4d2dff) into main (338d2b2) will decrease coverage by 0.23%.
Report is 2 commits behind head on main.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main     #159      +/-   ##
==========================================
- Coverage   38.82%   38.60%   -0.23%     
==========================================
  Files          31       31              
  Lines        2246     2259      +13     
==========================================
  Hits          872      872              
- Misses       1325     1338      +13     
  Partials       49       49

Flag	Coverage Δ
unittests	`38.60% <0.00%> (-0.23%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
pkg/ebpf/tracer.go	`0.00% <0.00%> (ø)`

bpf/rtt_tracker.h

dushyantbehl · 2023-07-27T17:30:17Z

/ok-to-test

github-actions · 2023-07-27T17:33:30Z

New image:
quay.io/netobserv/netobserv-ebpf-agent:3f822c7

It will expire after two weeks.

To deploy this build, run from the operator repo, assuming the operator is running:

USER=netobserv VERSION=3f822c7 make set-agent-image

github-actions · 2023-07-28T09:08:36Z

New image:
quay.io/netobserv/netobserv-ebpf-agent:0ffaeef

It will expire after two weeks.

To deploy this build, run from the operator repo, assuming the operator is running:

USER=netobserv VERSION=0ffaeef make set-agent-image

openshift-ci-robot · 2023-07-28T12:40:24Z

@dushyantbehl: This pull request references NETOBSERV-1112 which is a valid jira issue.

In response to this:

In a client to server communication, client sends SYN, server sends ACK. The RTT is calculated when ACK arrives and is reported on the ingress flow from Server->Client as the ACK packet corresponds to that flow.
This patch reverses the flow direction and finds the Client -> Server flow entry and adds RTT to the aggregated metrics.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

msherif1234 · 2023-07-31T11:40:02Z

bpf/types.h

 // No need to emit this struct. It's used only in kernel space
 typedef struct flow_seq_id_t {
    u16 src_port;
    u16 dst_port;
    u8 src_ip[IP_MAX_LEN];
    u8 dst_ip[IP_MAX_LEN];
    u32 seq_id;
+    u8 transport_protocol;
+    u32 if_index; // OS interface index
+    u8 __padding;


no need to add manual padding attribute will takecare of it, also seq_id is becoming more like flow_id I feel we can use the global hash map to calculate RTT directly none tcp fields like icmp ones can be set to 0

I added transport_protocol to make it future proof, to address your comment of seq id collision. and to make it similar to the dns seq_id to combine them in future if needed.
Padding is something I can remove.
if index is needed because flow agent connects to many interfaces in the same path which can affect time stamp when SYN and ACK were recorded to a. large extent if the same flow passes through eth0 -> bridge -> container veth -> container namespace ethernet etc

yeah its just more and more becoming similar to flow_id

There won't be more additions to it in my opinion. This map also stores seq_id which is the primary identifier here which does not have to be part of flow_id.

msherif1234 · 2023-07-31T11:41:51Z

pkg/ebpf/tracer.go

@@ -101,6 +101,8 @@ func NewFlowFetcher(cfg *FlowFetcherConfig) (*FlowFetcher, error) {
 	if enableRtt == 0 {
 		// Cannot set the size of map to be 0 so set it to 1.
 		spec.Maps[flowSequencesMap].MaxEntries = uint32(1)
+	} else {
+		log.Infof("RTT calculations are enabled")


is this for debugging ? we can know what is enabled or not from looking at the configs env var correct ?

Was just meant as info to see this via logs yeah...can be removed safely.

only problem I see is that if both directions (Ingress and Egress are not enabled) RTT calculations will be disabled and message will be present only in logs.

is it even possible to disable tc on one side but not the other ?

well do you mean tc queues? I am not sure but don't think its possible.
What I had meant is if ebpf agent somehow is set to attach only at one of Ingress or Egress, the DIRECTION configuration for agent. I believe right now this option is not exposed to the operator so we shouldn't worry too much.

openshift-ci-robot · 2023-08-01T13:34:06Z

@dushyantbehl: This pull request references NETOBSERV-1112 which is a valid jira issue.

In response to this:

The calculation of RTT in ebpf agent was initially implemented to be doing something like this,

We were storing the timestamp of outgoing SYN and incoming ACK, calculating the RTT as Incoming ACK - Outgoing SYN.

This works for calculations done on both client and server endpoints as both have one SYN which is sent out and one ack which is received in response to that to start a TCP connection.

ebpf agent on the other hand does not connect only at a single interface in Openshift environment. If we look at the same communication being done from inside a container it would look like this.

In this case the container contains a veth pair and ebpf agent attaches to the veth interface in the root namespace, let's call veth for a vpeer in the container namespace.
Any packet emerging from vpeer which is put at the egress queue of container will come at the ingress queue of the veth interface. Hence if we try to calculate RTT using the previous logic of outgoing SYN and incoming ACK then we would end up calculating the timestamp of SYN(Y) and ACK(Y) which will be just the response time of the container and not a correct value of the RTT.

Also note that the ebpf agent attaches to many interfaces in Openshift and can attach to interfaces like eth0 which always act like a Client or Server endpoints and hence the RTT calculation of Incoming ACK - Outgoing SYN will work for such endpoints.

So to account for both the cases we changed the RTT calculation in a simple way, we simply take the RTT of SYN/SYN+ACK/ACK separately, i.e. we will calculate RTT of SYN(X) and ACK(X) and SYN(Y) and ACK(Y) and simply take the maximum of both values.
This is needed because ebpf agent attaches at both type of interfaces, root veth interfaces which act like middleboxes and eth0 interfaces which act like final connection endpoints.

Note that the flows are in reverse direction so we need to calculate RTT for reverse flows in ebpf agent but its pretty easy to do.
Also, this calculation is interface specific so we need to add interface id to the sequence identifier.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

msherif1234 · 2023-08-01T16:26:27Z

/lgtm

jotak · 2023-08-02T14:50:48Z

@dushyantbehl it might be a naive question, but the problem that it is solving, couldn't it be "just" solved by matching the interface on ACK? (ie. making sure the ACK we get is on the same interface as the initial SYN) ? I guess by adding the interface as a seq map values?

jotak

@dushyantbehl @msherif1234 Was it tested end to end in ocp?
If so, I'm fine to merge if you're both ok with it (I just had a comment, wondering if there was a simpler approach possible - as this is a quite different algorithm now)

jotak · 2023-08-02T14:11:23Z

bpf/rtt_tracker.h

+    dst->if_index = src->if_index;
+
+    // Fields which should be reversed
+    dst->direction = (src->direction == INGRESS) ? EGRESS : INGRESS;


not a big deal but:

Suggested change

dst->direction = (src->direction == INGRESS) ? EGRESS : INGRESS;

dst->direction = 1 - src->direction;

I did an end to end testing at my level and @jpinsonneau and @ronensc (who reported the issue at their end) did the testing too.

on your suggestion, I fear if we write it like that and in future change INGRESS and EGRESS to be something other than 0 and 1 then the trick might fail silently, hence maybe better to write it explictily.

dushyantbehl · 2023-08-03T07:10:22Z

@dushyantbehl it might be a naive question, but the problem that it is solving, couldn't it be "just" solved by matching the interface on ACK? (ie. making sure the ACK we get is on the same interface as the initial SYN) ? I guess by adding the interface as a seq map values?

@jotak yes what you suggest is also part of the solution to match interface when matching SYN and ACK and it is done by adding interface to the seq_id map structure.

netobserv-ebpf-agent/bpf/types.h

Line 133 in 5f9a07f

u32 if_index; // OS interface index

jotak · 2023-08-03T09:02:40Z

I have the feeling that this new algorithm will still cause troubles especially when there is sampling. When you say:

we will calculate RTT of SYN(X) and ACK(X) and SYN(Y) and ACK(Y) and simply take the maximum of both values.

That works when we capture all the flows, because there is either "SYN(X) / ACK(X)" or "SYN(Y) / ACK(Y)" that corresponds to the actual RTT, whereas the other one is just process time and must be ignored. So taking the biggest should, in general, match the RTT and that's fine... But that doesn't work with sampling if you don't capture the flows that are relevant for RTT, then you would store the wrong value, which could screw up all the metrics computed from that.

So I have the feeling we need to find something else.

It sounds like the root cause of these problem is that we're never sure what INGRESS or EGRESS really mean (and we solved that problem in FLP with something called "reinterpret_direction", but we can't have the same solution here, we need to find something else). I'm not sure how we could do that. Maybe by reversing the meaning of INGRESS and EGRESS depending on which interface we're on?

Alternatively maybe we should restrict on which interfaces we are computing RTT, e.g. in your graph above we would use only eth0 and ignore veth. The downside is that it decreases the chances to get the RTT when sampling is on.

wdyt?

dushyantbehl · 2023-08-03T09:58:03Z

I have the feeling that this new algorithm will still cause troubles especially when there is sampling. When you say:

we will calculate RTT of SYN(X) and ACK(X) and SYN(Y) and ACK(Y) and simply take the maximum of both values.

That works when we capture all the flows, because there is either "SYN(X) / ACK(X)" or "SYN(Y) / ACK(Y)" that corresponds to the actual RTT, whereas the other one is just process time and must be ignored. So taking the biggest should, in general, match the RTT and that's fine... But that doesn't work with sampling if you don't capture the flows that are relevant for RTT, then you would store the wrong value, which could screw up all the metrics computed from that.

This flow makes 3 packets, 1) SYN(X) 2) SYN(Y) ACK(X) 3) ACK(Y)
if by any chance of SAMPLING we miss only one packet out of 1 and 2 then the calculation could be wrong and provide a very small value. This can be fixed by having a check at some point in the pipeline to not show values lower than some micro seconds.

SAMPLING the way it is done right now interacts with all packets so we have to implement a workaround or change SAMPLING for this.

So I have the feeling we need to find something else.

It sounds like the root cause of these problem is that we're never sure what INGRESS or EGRESS really mean (and we solved that problem in FLP with something called "reinterpret_direction", but we can't have the same solution here, we need to find something else). I'm not sure how we could do that. Maybe by reversing the meaning of INGRESS and EGRESS depending on which interface we're on?

We cannot directly determine from ebpf which type of interface we are connected on, a veth interface could either be vpeer or veth from the diagram above. This problem is unique here because ebpf agent connects to all possible interfaces in the system hence we need to look at both side of flows.

Alternatively maybe we should restrict on which interfaces we are computing RTT, e.g. in your graph above we would use only eth0 and ignore veth. The downside is that it decreases the chances to get the RTT when sampling is on.

There will be a high chance to miss some flows in the system if we take this approach in my opinion. Traffic from one pod on one host to another pod on a different host is encapsulated in host IP so if we only calculate at interfaces like eth0 we will only get host ip addresses as part of flow and not the actual pod IP which we might be interested in.

wdyt?

msherif1234 · 2023-08-03T12:11:16Z

at somepoint I was working on new hook that just triggered when tcp state change independent off tc hook https://github.com/netobserv/netobserv-ebpf-agent/pull/106/files#diff-f256390f160e8c3dfdbea43540fc179790fed2b28286efdf60bea2a4711f26c1R53 I feel that would get us out of the issue with sampling and missing state update ?

I am planning to bring that PR back to life at somepoint as its quiet out of date but instead of generating events we will just create flows in the hashmap

jotak · 2023-08-03T12:46:20Z

@dushyantbehl

if by any chance of SAMPLING we miss only one packet out of 1 and 2 then the calculation could be wrong and provide a very small value. This can be fixed by having a check at some point in the pipeline to not show values lower than some micro seconds.

So in that case I think this would be on the agent to not show values of microseconds magnitude, rather than later in the pipeline: because that's due to implementation details of the agent, and the other components should be able to trust what the agent sends.

We cannot directly determine from ebpf which type of interface we are connected on, a veth interface could either be vpeer or veth from the diagram above. This problem is unique here because ebpf agent connects to all possible interfaces in the system hence we need to look at both side of flows.

Maybe we should try asking around ... ovn, ovs folks might have an idea that we're missing, about how we could solve that? We get the if_index right, maybe we can do something based on that?

@msherif1234 yeah that would be an interesting thing to try, I agree!

openshift-ci-robot · 2023-08-03T14:46:10Z

@dushyantbehl: This pull request references NETOBSERV-1112 which is a valid jira issue.

In response to this:

The calculation of RTT in ebpf agent was initially implemented to be doing something like this,

We were storing the timestamp of outgoing SYN and incoming ACK, calculating the RTT as Incoming ACK - Outgoing SYN.

This works for calculations done on both client and server endpoints as both have one SYN which is sent out and one ack which is received in response to that to start a TCP connection.

ebpf agent on the other hand does not connect only at a single interface in Openshift environment. If we look at the same communication being done from inside a container it would look like this.

In this case the container contains a veth pair and ebpf agent attaches to the veth interface in the root namespace, let's call veth for a vpeer in the container namespace.
Any packet emerging from vpeer which is put at the egress queue of container will come at the ingress queue of the veth interface. Hence if we try to calculate RTT using the previous logic of outgoing SYN and incoming ACK then we would end up calculating the timestamp of SYN(Y) and ACK(Y) which will be just the response time of the container and not a correct value of the RTT.

Also note that the ebpf agent attaches to many interfaces in Openshift and can attach to interfaces like eth0 which always act like a Client or Server endpoints and hence the RTT calculation of Incoming ACK - Outgoing SYN will work for such endpoints.

So to account for both the cases we changed the RTT calculation in a simple way, we simply take the RTT of SYN/SYN+ACK/ACK separately, i.e. we will calculate RTT of SYN(X) and ACK(X) and SYN(Y) and ACK(Y) and simply take the maximum of both values.
This is needed because ebpf agent attaches at both type of interfaces, root veth interfaces which act like middleboxes and eth0 interfaces which act like final connection endpoints.

Note that the flows are in reverse direction so we need to calculate RTT for reverse flows in ebpf agent but its pretty easy to do.
Also, this calculation is interface specific so we need to add interface id to the sequence identifier.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

dushyantbehl · 2023-08-03T14:52:20Z

@dushyantbehl

if by any chance of SAMPLING we miss only one packet out of 1 and 2 then the calculation could be wrong and provide a very small value. This can be fixed by having a check at some point in the pipeline to not show values lower than some micro seconds.

So in that case I think this would be on the agent to not show values of microseconds magnitude, rather than later in the pipeline: because that's due to implementation details of the agent, and the other components should be able to trust what the agent sends.

Actually I feel that doing this filtering at agent will be much more hardcoded behavior, a very simple fix is to mark in SAMPLING to not throw away SYN/ACK packets which are needed for many other behavior as well.

We cannot directly determine from ebpf which type of interface we are connected on, a veth interface could either be vpeer or veth from the diagram above. This problem is unique here because ebpf agent connects to all possible interfaces in the system hence we need to look at both side of flows.

Maybe we should try asking around ... ovn, ovs folks might have an idea that we're missing, about how we could solve that? We get the if_index right, maybe we can do something based on that?

This is not doable based on if_index or anything else, it just gives name and id of the interface while the direction of capture depends on where the traffic is coming from. With the current patch only 2 packets are being looked at and RTT is being calculated based on them which is the standard approach.

jotak · 2023-08-03T16:08:42Z

bpf/configs.h

@@ -6,5 +6,6 @@
 volatile const u32 sampling = 0;
 volatile const u8 trace_messages = 0;
 volatile const u8 enable_rtt = 0;
+volatile const u64 min_rtt = 50000; //50 micro seconds


Thanks @dushyantbehl ! this will avoid having inconsistent values. We can iterate later to find a better solution. I don't despair of finding a good solution closer to your first implementation, which I found simpler but would need a twist, as you put it in this PR description :-)

If that's good for @msherif1234 , that's good for me too

Thanks @jotak

msherif1234 · 2023-08-04T10:55:41Z

bpf/configs.h

@@ -6,5 +6,6 @@
 volatile const u32 sampling = 0;
 volatile const u8 trace_messages = 0;
 volatile const u8 enable_rtt = 0;
+volatile const u64 min_rtt = 50000; //50 micro seconds


are u planning to make this configurable ? because I don't see the userspace pieces to do that if not then u can define in it flows.c where its actually used and no need to make volatile in that case

msherif1234 · 2023-08-04T10:58:27Z

bpf/rtt_tracker.h

+    u64 *prev_ts = (u64 *)bpf_map_lookup_elem(&flow_sequences, seq_id);
+    if (prev_ts != NULL) {
+        u64 rtt = pkt->current_ts - *prev_ts;
+        // Because of SAMPLING the way it is done if we miss one of SYN/SYN+ACK/ACK


I will also add FIXME here and indicate this a temp workaround for this case till we have better solution

Makes sense to me. And since this is fixme I would make the above variable non volatile.

Signed-off-by: Dushyant Behl <dushyantbehl@hotmail.com> Signed-off-by: Dushyant Behl <dushyantbehl@users.noreply.github.com>

msherif1234 · 2023-08-04T12:06:37Z

/lgtm

dushyantbehl · 2023-08-04T15:06:43Z

/approve

openshift-ci · 2023-08-04T15:06:48Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dushyantbehl

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [dushyantbehl]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

dushyantbehl added the do-not-merge/hold label Jul 20, 2023

openshift-ci bot added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jul 20, 2023

msherif1234 reviewed Jul 20, 2023

View reviewed changes

bpf/rtt_tracker.h Outdated Show resolved Hide resolved

openshift-merge-robot added the needs-rebase label Jul 21, 2023

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jul 26, 2023

dushyantbehl force-pushed the main branch from 5f18ed9 to 9898807 Compare July 27, 2023 16:59

openshift-ci bot added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jul 27, 2023

Make RTT values show up on both flow directions

35fc676

dushyantbehl force-pushed the main branch from 9898807 to e3999da Compare July 28, 2023 09:06

openshift-merge-robot removed the needs-rebase label Jul 28, 2023

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jul 28, 2023

dushyantbehl added bug Something isn't working ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. and removed do-not-merge/hold labels Jul 28, 2023

dushyantbehl changed the title ~~RTT latency is visible in one direction, This PR adds RTT measurement to both direction flows.~~ NETOBSERV-1112: This patch fixes a bug where RTT was not visible for flow logs at times. Jul 28, 2023

openshift-ci-robot added the jira/valid-reference label Jul 28, 2023

msherif1234 reviewed Jul 31, 2023

View reviewed changes

dushyantbehl force-pushed the main branch from e3999da to 5f9a07f Compare August 1, 2023 12:48

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Aug 1, 2023

openshift-ci bot assigned msherif1234 Aug 1, 2023

openshift-ci bot added the lgtm label Aug 1, 2023

jotak reviewed Aug 2, 2023

View reviewed changes

dushyantbehl closed this Aug 3, 2023

dushyantbehl reopened this Aug 3, 2023

dushyantbehl force-pushed the main branch from 5f9a07f to 09194dc Compare August 3, 2023 16:00

openshift-ci bot removed the lgtm label Aug 3, 2023

jotak reviewed Aug 3, 2023

View reviewed changes

msherif1234 reviewed Aug 4, 2023

View reviewed changes

Fix RTT calculation error for container hooks.

e4d2dff

Signed-off-by: Dushyant Behl <dushyantbehl@hotmail.com> Signed-off-by: Dushyant Behl <dushyantbehl@users.noreply.github.com>

dushyantbehl force-pushed the main branch from 09194dc to e4d2dff Compare August 4, 2023 12:03

openshift-ci bot added the lgtm label Aug 4, 2023

openshift-ci bot added the approved label Aug 4, 2023

openshift-merge-robot merged commit 6d9d2e7 into netobserv:main Aug 4, 2023
9 checks passed

	dst->direction = (src->direction == INGRESS) ? EGRESS : INGRESS;
	dst->direction = 1 - src->direction;

NETOBSERV-1112: This patch fixes a bug where RTT was not visible for flow logs at times. #159

NETOBSERV-1112: This patch fixes a bug where RTT was not visible for flow logs at times. #159

Conversation

dushyantbehl commented Jul 20, 2023 • edited Loading

dushyantbehl commented Jul 20, 2023

github-actions bot commented Jul 20, 2023

codecov bot commented Jul 20, 2023 • edited Loading

Codecov Report

dushyantbehl commented Jul 27, 2023

github-actions bot commented Jul 27, 2023

github-actions bot commented Jul 28, 2023

openshift-ci-robot commented Jul 28, 2023 • edited by openshift-ci bot Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openshift-ci-robot commented Aug 1, 2023 • edited by openshift-ci bot Loading

msherif1234 commented Aug 1, 2023

jotak commented Aug 2, 2023

jotak left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dushyantbehl commented Aug 3, 2023

jotak commented Aug 3, 2023 • edited Loading

dushyantbehl commented Aug 3, 2023

msherif1234 commented Aug 3, 2023

jotak commented Aug 3, 2023

openshift-ci-robot commented Aug 3, 2023 • edited by openshift-ci bot Loading

dushyantbehl commented Aug 3, 2023

jotak Aug 3, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msherif1234 commented Aug 4, 2023

dushyantbehl commented Aug 4, 2023

openshift-ci bot commented Aug 4, 2023

dushyantbehl commented Jul 20, 2023 •

edited

Loading

codecov bot commented Jul 20, 2023 •

edited

Loading

openshift-ci-robot commented Jul 28, 2023 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Aug 1, 2023 •

edited by openshift-ci bot

Loading

jotak commented Aug 3, 2023 •

edited

Loading

openshift-ci-robot commented Aug 3, 2023 •

edited by openshift-ci bot

Loading

jotak Aug 3, 2023 •

edited

Loading