diff --git a/blogs/index.md b/blogs/index.md index 88df331..1428635 100644 --- a/blogs/index.md +++ b/blogs/index.md @@ -7,3 +7,4 @@ - [Real-Time Per Flow Packets Drop](./packet_drops/index.md) - [Secondary interface support](./secondary_interface/index.md) - [What's New in Network Observability 1.4](./whats_new_1.4/index.md) +- [TCP Round Trip Time tracking](./rtt/index.md) diff --git a/blogs/rtt/images/rtt_graphs.png b/blogs/rtt/images/rtt_graphs.png index cbf2c64..83f05cf 100644 Binary files a/blogs/rtt/images/rtt_graphs.png and b/blogs/rtt/images/rtt_graphs.png differ diff --git a/blogs/rtt/index.md b/blogs/rtt/index.md new file mode 100644 index 0000000..4ca4db5 --- /dev/null +++ b/blogs/rtt/index.md @@ -0,0 +1,160 @@ +# Network Observability TCP Handshake Round Trip Time + +By: Dushyant Behl, Julien Pinsonneau and Mohamed S. Mahmoud + +In OpenShift Container Platform (OCP), ensuring efficient packet delivery is +paramount for maintaining seamless communication between applications. +However, challenges like network congestion, misconfigured systems, +or hardware limitations can lead to slow connections, impacting overall +performance. Round Trip Time (RTT), typically measured in milliseconds, +plays a crucial role in monitoring network health and diagnosing issues. + +## Implementing Smooth Round-Trip Time (SRTT) with eBPF + +The RTT is the time it takes for a packet to travel from the sender to the receiver +and back. In a network, RTT can vary due to factors like network congestion, +varying route lengths, and other dynamic conditions. +SRTT is introduced to provide a more consistent and less jittery representation +of the RTT. + +In TCP, RTT is a crucial metric. + +Our implementation leverages eBPF to register to `fentry` eBPF hook +for `tcp_rcv_established()`. +We extract the SRTT (smooth round-trip time) value from TCP sockets, correlating it +to existing flows and enriching them with RTT values in nanoseconds. + +When a new Netobserv flow is created, and the RTT (Round-Trip Time) feature is enabled, +an initial RTT of `10usec` is assigned. +This initial value for RTT may be considered quite low. + +Upon triggering the eBPF (Extended Berkeley Packet Filter) socket, the flow RTT +value is updated to reflect the maximum RTT value for that specific flow. + +For more detailed explanation of smoothed RTT estimation, refer to [Karn's algorithm paper](http://ccr.sigcomm.org/archive/1995/jan95/ccr-9501-partridge87.pdf) + +![TCP based RTT calculations](./images/tcp_rtt_calculations.png) + +### Why using `fentry` eBPF hook + +The eBPF `fentry` programs have lower overhead as they trigger +the hook before calling the kernel function of interest. + +In our implementation: + +1. Register and link `fentry` hook for kernel's `tcp_rcv_established()` + +```cgo +SEC("fentry/tcp_rcv_established") +int BPF_PROG(tcp_rcv_fentry, struct sock *sk, struct sk_buff *skb) { + if (sk == NULL || skb == NULL) { + return 0; + } + return calculate_flow_rtt_tcp(sk, skb); +} +``` + +2. Reconstruct the Netobserv flow key, including incoming interface Layer2, Layer3, and Layer4 info. + +3. Match existing flows in the PerCPU hashmap flow table and enrich them with srtt info from TCP sockets. If + multiple SRTT values exist for the same flow, we take the maximum value. + +Currently, our approach calculates RTT only for the TCP packets so flows which are non-TCP do not show RTT information. + +## Potential Use Cases + +Flow RTT capture from eBPF `flow_monitor` hookpoint can serve various purposes: + +- Network Monitoring: Gain insights into TCP handshakes, helping + network administrators identify unusual patterns, potential bottlenecks, or + performance issues. + +- Troubleshooting: Debug TCP-related issues by tracking latency and identifying + misconfigurations. + +## How to Enable RTT + +To enable this feature we need to create a FlowCollector object with the following +fields enabled in eBPF config section as below: + +```yaml +apiVersion: flows.netobserv.io/v1beta2 +kind: FlowCollector +metadata: + name: cluster +spec: + agent: + type: eBPF + ebpf: + features: + - FlowRTT +``` + +## A Quick Tour in the UI + +Once the `FlowRTT` feature is enabled, the OCP console plugin automatically adapts +to provide additional filter and show information across Netflow Traffic page views. + +Open your OCP Console and move to +`Administrator view` -> `Observe` -> `Network Traffic` page as usual. + +A new filter, `Flow RTT` is available in the common section: + +![flow rtt filter](./images/flow_rtt_filter.png) + +The `FlowRTT` filter will allow you to capture any flow that has an RTT more than a specific time in nanoseconds. + +For production users, filtering on the `TCP` protocol, `Ingress` direction, +and looking for `FlowRTT` values greater than 10,000,000 nanoseconds (10ms) +can help identify TCP flows with high latency. +This filtering approach allows users to focus on specific network flows that may +be experiencing significant delays. +By setting a threshold of `10ms`, you can efficiently isolate and address potential +latency issues in your TCP traffic. + +### Overview + +New graphs are introduced in the `Advanced options` -> `Manage panels` popup: + +![advanced options](./images/advanced_options.png) + +- Top X average TCP handshake Round Trip Time with overall (donut or lines) +- Bottom X minimum TCP handshake Round Trip Time with overall (donut or lines) +- Top X maximum TCP handshake Round Trip Time with overall (donut or lines) +- Top X 90th percentile TCP handshake Round Trip Time with overall (donut or lines) +- Top X 99th percentile TCP handshake Round Trip Time with overall (donut or lines) +- +![rtt graphs](./images/rtt_graphs.png) + +These two graphs can help you to identify the slowest TCP flows and their trends +over time. Use the filters to drill down into specific pods, namespaces or nodes. + +### Traffic flows + +The table view shows the `Flow RTT` in both column and side panel. + +![rtt table](./images/rtt_table.png) + +### Topology + +Last but not least, the topology view displays min / max / avg / p90 / p99 `RTT` +latency on edges. +Clicking on a node or an edge will allow you to see per direction metrics and +the related graph. + +![rtt topology](./images/rtt_topology.png) + +### Future improvments + +Here is a non exhaustive list of future improvements coming for a full featured +Round Trip Time analysis: + +- Latest RTT in topology view +- Prometheus metrics and alerting + +## Feedback + +We hope you liked this article ! + +Netobserv is an OpenSource project [available on github](https://github.com/netobserv). +Feel free to share your [ideas](https://github.com/netobserv/network-observability-operator/discussions/categories/ideas), [use cases](https://github.com/netobserv/network-observability-operator/discussions/categories/show-and-tell) or [ask the community for help](https://github.com/netobserv/network-observability-operator/discussions/categories/q-a). diff --git a/blogs/rtt/round_trip_time.md b/blogs/rtt/round_trip_time.md deleted file mode 100644 index 9226c0a..0000000 --- a/blogs/rtt/round_trip_time.md +++ /dev/null @@ -1,140 +0,0 @@ -# Network Observability TCP Handshake Round Trip Time Developer Preview - -![logo](./images/rtt_logo.png) - -By: Dushyant Behl and Julien Pinsonneau - -In OCP, ensuring efficient packet delivery is crucial for maintaining smooth -communication between applications. However, due to various factors such -as network congestion, misconfigured systems, or hardware limitations, -connections may become slow, impacting overall performance. -Round Trip Time (RTT) is the duration, usually measured in milliseconds, from when -a client sends a request to when it receives a response from a server. -Mesuring RTT is important for monitoring network health and diagnosing issues. - -## RTT calculation using eBPF - -RTT calculations between a server and a client requires sending a packet from the -server to the client and then back from client to the server. -RTT is then calculated as the time measured between sending and receiving the packet back. -RTT calculations gives a very good estimate on how the network is behaving in between -two hops. In netobsev ebpf-agent, we use the same idea for calculating RTT but we do it -for TCP connections which is also called TCP Based RTT calculations. - -![TCP based RTT calculations](./images/tcp_rtt_calculations.png) - -In TCP, as shown in the diagram above, at the start of the connection, -a client sends a SYN packet to the server with a random sequence number (X) and the -server responds with an ACK (X+1), along with the ACK, the server also sends a SYN -packet of its own with a different sequence number (Y) to which client resonds -with an ack (Y+1). This is what is called a three-way-handshake for establishing a -TCP connection and we use these SYN/ACK packet pairs at the start of the connection to -calculate RTT between the server and the client. - -In our implementation, - -1. ebpf-agent keeps track of the SYN/ACK packet pairs crossing through the agent at the start of every TCP connection. - -2. For every SYN packet that gets detected at the `flow_monitor` hookpoint, the agent will capture standard 4-tuple information (src/dst ip addresses and ports), packet sequence identifier and the interface it was detected on and saves it into a `flow_sequences` hashmap along with the timestamp of the packet detection. - -3. Now for every ACK packet that gets detected by the agent at the `flow_monitor` hookpoint, it checks by reversing the flow direction, if the 4-tuple information, the identifier (sequence id of ACK - 1) along with the interface id, is present in the `flow_sequences` hashmap, if so it will calculate the handshake RTT as, -`rtt = ack-timestampack - syn-timestamp(from map)` - -We present the calculated RTT as part of the flow log information exported by the agent. Agent will report actual RTT for the flow if it is calculated and present (for tcp handshake packets), zero if it is not calculated (for non handshake packets or any other protocol) - -Please note that the RTT calculations can be done on both the endpoints (client and server) as both send a SYN and receive an ACK in the 3-way TCP handshake. - -Currently our approcah calculates RTT only for the TCP handshakes packets so flows which are non TCP will not show RTT information. -Also, in the ebpf-agent we implement a sampling strategy which keeps one out of n (configurable) packets in the flow, so the design of RTT calculations currently require a high sampling rate to be used as a low sampling strategy can miss either SYN, ACK or both packets leading to no calculation of RTT. - -We plan to implement a different sampling strategy to bypass this issue and implement support for more protocols in the future. - -## Potential Use Cases - -Flow RTT capture from eBPF `flow_monitor` hookpoint can serve various purposes: - -- Network Monitoring: Gain insights into TCP handshakes, helping - network administrators identify unusual patterns, potential bottlenecks, or - performance issues. - -- Troubleshooting: Debug TCP-related issues by tracking latency and identifying - misconfigurations. - -## How to enable RTT - -To enable this feature we need to create a flow collector object with the following -fields enabled in eBPF config section as below: - -```yaml -apiVersion: flows.netobserv.io/v1beta1 -kind: FlowCollector -metadata: - name: cluster -spec: - agent: - type: EBPF - ebpf: - sampling: 1 - features: - - FlowRTT -``` - -Sampling 1 will ensure we capture both SYN and ACK packets to calculate RTT. - -## A quick tour in the UI - -Once the `FlowRTT` feature is enabled, the OCP console plugin automatically adapts -to provide additional filter and show information across Netflow Traffic page views. - -Open your OCP Console and move to -`Administrator view` -> `Observe` -> `Network Traffic` page as usual. - -A new filter, `Flow RTT` is available in the common section: - -![flow rtt filter](./images/flow_rtt_filter.png) - -It will allow you capture any flow having a RTT more than a specific time in nanoseconds. - -### Overview - -New graphs are introduced in the `advanced options` -> `manage panels` popup: - -![advanced options](./images/advanced_options.png) - -- Top X average flow RTT (donut) -- Top X average flow RTT (lines) - -![rtt graphs](./images/rtt_graphs.png) - -These two graphs will help you to identify the slowest TCP flows and their trends -over time. Use the filters to drill down into specific pods, namespaces or nodes. - -### Traffic flows - -The table view shows the `Flow RTT` in both column and side panel. - -![rtt table](./images/rtt_table.png) - -### Topology - -Last but not least, the topology view displays average `RTT` latency on edges. -Clicking on a node or an edge will allow you to see per direction metrics and -the related graph. - -![rtt topology](./images/rtt_topology.png) - -### Future improvments - -Here is a non exhaustive list of future improvements coming for a full featured -Round Trip Time analysis: - -- Max / P90 / P99 graphs -- Latest / Max RTT in topology view -- Prometheus metrics and alerting - -## Feedback - -We hope you liked this article ! - -Netobserv is an OpenSource project [available on github](https://github.com/netobserv). -Feel free to share your [ideas](https://github.com/netobserv/network-observability-operator/discussions/categories/ideas), [use cases](https://github.com/netobserv/network-observability-operator/discussions/categories/show-and-tell) or [ask the community for help](https://github.com/netobserv/network-observability-operator/discussions/categories/q-a).