NETOBSERV-1248 deduper using infinispan cache #639

jpinsonneau · 2024-03-19T09:14:55Z

Description

Implements infinispan shared cache:

Add cache functions for deduper case
Keeps Egress flows between two nodes in cache
Then merge interfaces and ifdirections when Ingress is captured
Make url configurable
Auth / tls
Implements extra cases

Merged flows contains a list of interfaces / ifdirections with a * next to names coming from Egress flow:

Steps to use this code:

go mod tidy && go mod vendor
Install Infinispan Operator OR Red Hat Data Grid Operator
Create cache cluster in netobserv namespace with disabled TLS / Auth

apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: cache
  namespace: netobserv
spec:
  security:
    endpointAuthentication: false
    endpointEncryption:
      clientCert: None
      type: None
    endpointSecretName: cache-generated-secret
  container:
    memory: 1Gi
  service:
    replicationFactor: 2
    type: Cache
  jmx: {}
  configListener:
    enabled: true
    logging:
      level: info
  upgrades:
    type: Shutdown
  version: 8.4.6-2
  replicas: 1

Create deduper cache

apiVersion: infinispan.org/v2alpha1
kind: Cache
metadata:
  name: deduper
  namespace: netobserv
spec:
  clusterName: cache
  name: deduper
  updates:
    strategy: retain

These will provide a service cache.netobserv.svc.cluster.local on port 11222.

Dependencies

n/a

Checklist

If you are not familiar with our processes or don't know what to answer in the list below, let us know in a comment: the maintainers will take care of that.

openshift-ci · 2024-03-19T09:15:01Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

openshift-ci · 2024-03-19T09:15:03Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from jpinsonneau. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

jpinsonneau · 2024-03-19T09:22:58Z

pkg/pipeline/transform/infinispan/cache.go

+
+func (c *Cache) getDedupKey(v config.GenericMap) string {
+	return regex.ReplaceAllString(
+		fmt.Sprintf("%d-%d-%s-%d-%s-%d", v["TimeFlowEndMs"], v["Proto"], v["SrcAddr"], v["SrcPort"], v["DstAddr"], v["DstPort"]),


I put TimeFlowEndMs on top of the 5 tuple to avoid conflicts when 2 Egress comes before first Ingress.
As is, I miss some flows as you can see in the screens. We need to find a better way to unique identify each flows.

Any thoughts ?

Perhaps we can do an assumption here, that the interfaces are not going to change over time across flows in the same 5-tuples. So we could remove the timestamp from the key. In fact I'm already doing this same assumption here
On top of that, it should be much better for the memory footprint.

The downside is that it works for merging interfaces, but might not work if we want to reuse the cache for other purposes.

this cache is only for deduper so I'm fine with that;

However are we sure that the interfaces are not changing when the pod is recreated for example ?
The ip addresses can be the same but ovn generated interfaces could change 🤔

on top of that, I could push only fields we want to merge in the cache instead of the whole flow

However are we sure that the interfaces are not changing when the pod is recreated for example ?
The ip addresses can be the same but ovn generated interfaces could change

A pod restart would very very very likely use a different IP as far as I know; and even if that's not the case, the connection would need to be reinitiated so the source port would change

but perhaps in any case there should be an expiry mechanism

the cache already expire when full 😸 you can also set a lifespan
https://infinispan.org/docs/stable/titles/configuring/configuring.html#expiration_configuring-memory-usage

I made changes in 335d737

jpinsonneau · 2024-03-19T09:23:41Z

pkg/pipeline/transform/infinispan/cache.go

+		Endpoint: "http://cache.netobserv.svc.cluster.local:11222",
+		/*Credentials: &Credentials{
+			Username: "netobserv",
+			Password: "netobserv",
+		},*/


Need to make these configurable and add TLS from secret

jpinsonneau · 2024-03-19T09:28:17Z

pkg/pipeline/transform/transform_network.go

+					log.Debugf("merging cached flow %v with %v ...", srcFlow, outputEntry)
+					if len(srcFlow["Interfaces"].([]interface{})) == len(srcFlow["IfDirections"].([]interface{})) {
+						for i := 0; i < len(srcFlow["Interfaces"].([]interface{})); i++ {
+							outputEntry["Interfaces"] = append(outputEntry["Interfaces"].([]string), fmt.Sprintf("%s*", srcFlow["Interfaces"].([]interface{})[i].(string)))
+							outputEntry["IfDirections"] = append(outputEntry["IfDirections"].([]int), int(srcFlow["IfDirections"].([]interface{})[i].(float64)))
+						}
+					}
+					log.Debugf("merged flow: %v", outputEntry)


👆 merging Egress flow content in Ingress one.
For now just merging Interfaces and IfDirections and adding a * next to the names.
We could imagine merging more feature related contents, TCP flags and also check Bytes / Packets amounts

so if Egress comes after Ingress, it will be ignored? Perhaps it's another reason to not have the timestamp in the key as discussed here

yeah currently if Egress comes after Ingress, it's pushed in the cache and never retreived.
Only Ingress will appear in storage and will not have merged fields.

An improvment could be to send first flow to the cache whatever direction is found and merge on the "second" one; but that will double the cache queries

jotak · 2024-03-20T09:47:30Z

pkg/pipeline/transform/infinispan/cache.go

+)
+
+var log = logrus.WithField("component", "transform.Network.Cache")
+var regex = regexp.MustCompile(`[^a-zA-Z0-9 ]+`)


is there forbidden characters in key that require this replaceAll ?

I had issues with my initial attempt either with . or : when I had standart ips + ports key
I can give another try to get rid of this regex and improve perfs indeed

also addressed in 335d737

jotak · 2024-03-20T11:55:38Z

pkg/pipeline/transform/transform_network.go

-			reinterpretDirection(outputEntry, &n.DirectionInfo)
+			isInner, isSrcReporter := reinterpretDirection(outputEntry, &n.DirectionInfo)
+			if n.cache != nil {
+				return n.cache.DedupFlows(isInner, isSrcReporter, outputEntry)


I was weirdly seeing no flow under layer=app filter and now I understand : this return breaks the rules loop processing :-)
It should be continue

This case push flow to cache so you don't need to manage extra transform rules on it right ?
The goal is actually to break the pipeline here.

When the second flow will hit this stage, the first flow is retreived from cache and other transform rules are applied such as layer

jotak · 2024-03-20T14:24:36Z

The new version is better but still increases the overall resource usage in my tests, although it's decrease for Loki & storage:
https://docs.google.com/spreadsheets/d/1qakBaK1dk_rERO30k1cSR4W-Nn0SXW4A3lqQ1sZC4rE/edit#gid=705512397

(By overall I mean: everything deployed in netobserv* namespaces - so that includes the infinispan deployment and loki)

CPU overall +146% (1.68)
Memory overall +30% (3GB)

Loki CPU -44% (0.035)
Loki memory -46% (652MB)
Loki storage -53%

So the benefit here is really on storage, and comes at a price in CPU and memory. We could run tests at a bigger scale to check if it's different.

Also one thing that seems wrong from my tests is that the captured traffic has dropped quite a lot, as if we were removing more flows than expected. That's to investigate.

jpinsonneau · 2024-04-09T16:32:08Z

Rebased this PR and added CacheEndpoint config to condition cache usage

openshift-merge-robot · 2024-04-20T19:17:20Z

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jpinsonneau · 2024-07-29T12:17:33Z

@jotak is this something we should still consider ?

I can replace the json Marshal / Unmarshal by fmt.Sprintf to improve FLP CPU usage but we will still have the infinispan resource impact 🤔
If you think it's not worth I will just close that PR.

openshift-ci bot added the do-not-merge/work-in-progress label Mar 19, 2024

openshift-merge-robot added the needs-rebase label Mar 19, 2024

jpinsonneau commented Mar 19, 2024

View reviewed changes

jotak added ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. and removed ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. labels Mar 19, 2024

jotak reviewed Mar 20, 2024

View reviewed changes

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Mar 20, 2024

jotak added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Mar 20, 2024

jotak reviewed Mar 20, 2024

View reviewed changes

jotak mentioned this pull request Mar 25, 2024

flowlogs pipeline memory consumption large clusters. #641

Open

jpinsonneau added 2 commits April 9, 2024 18:30

vendor

f560edf

deduper using infinispan cache

1ccd91c

jpinsonneau force-pushed the infinispan_cache branch from 335d737 to 1ccd91c Compare April 9, 2024 16:31

openshift-merge-robot removed the needs-rebase label Apr 9, 2024

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Apr 9, 2024

jpinsonneau marked this pull request as ready for review April 9, 2024 16:31

openshift-ci bot removed the do-not-merge/work-in-progress label Apr 9, 2024

openshift-merge-robot added the needs-rebase label Apr 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NETOBSERV-1248 deduper using infinispan cache #639

NETOBSERV-1248 deduper using infinispan cache #639

jpinsonneau commented Mar 19, 2024 •

edited

Loading

openshift-ci bot commented Mar 19, 2024

openshift-ci bot commented Mar 19, 2024

jpinsonneau Mar 19, 2024

jotak Mar 20, 2024 •

edited

Loading

jpinsonneau Mar 20, 2024

jpinsonneau Mar 20, 2024

jotak Mar 20, 2024

jotak Mar 20, 2024

jpinsonneau Mar 20, 2024

jpinsonneau Mar 20, 2024

jpinsonneau Mar 19, 2024

jpinsonneau Mar 19, 2024

jotak Mar 20, 2024

jpinsonneau Mar 20, 2024

jotak Mar 20, 2024

jpinsonneau Mar 20, 2024

jpinsonneau Mar 20, 2024

jotak Mar 20, 2024

jpinsonneau Apr 9, 2024

jotak commented Mar 20, 2024

jpinsonneau commented Apr 9, 2024

openshift-merge-robot commented Apr 20, 2024

jpinsonneau commented Jul 29, 2024

NETOBSERV-1248 deduper using infinispan cache #639

Are you sure you want to change the base?

NETOBSERV-1248 deduper using infinispan cache #639

Conversation

jpinsonneau commented Mar 19, 2024 • edited Loading

Description

Dependencies

Checklist

openshift-ci bot commented Mar 19, 2024

openshift-ci bot commented Mar 19, 2024

Choose a reason for hiding this comment

jotak Mar 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jotak commented Mar 20, 2024

jpinsonneau commented Apr 9, 2024

openshift-merge-robot commented Apr 20, 2024

jpinsonneau commented Jul 29, 2024

jpinsonneau commented Mar 19, 2024 •

edited

Loading

jotak Mar 20, 2024 •

edited

Loading