Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NETOBSERV-627: merge Agent and FLP #502

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

NETOBSERV-627: merge Agent and FLP #502

wants to merge 4 commits into from
+1,294 −1,756

Conversation

jotak
Copy link
Member

@jotak jotak commented Nov 27, 2023 •
edited
Loading

Description

Do not deploy FLP: instead, configure the Agent to start FLP in-process. This only works without Kafka.

This is a PoC: consider merging only if it improves performances. It does

Dependencies

n/a

Checklist

If you are not familiar with our processes or don't know what to answer in the list below, let us know in a comment: the maintainers will take care of that.

  • Is this PR backed with a JIRA ticket? If so, make sure it is written as a title prefix (in general, PRs affecting the NetObserv/Network Observability product should be backed with a JIRA ticket - especially if they bring user facing changes).
  • Does this PR require product documentation?
    • If so, make sure the JIRA epic is labelled with "documentation" and provides a description relevant for doc writers, such as use cases or scenarios. Any required step to activate or configure the feature should be documented there, such as new CRD knobs.
  • Does this PR require a product release notes entry?
    • If so, fill in "Release Note Text" in the JIRA.
  • Is there anything else the QE team should know before testing? E.g: configuration changes, environment setup, etc.
    • If so, make sure it is described in the JIRA ticket.
  • QE requirements (check 1 from the list):
    • Standard QE validation, with pre-merge tests unless stated otherwise.
    • Regression tests only (e.g. refactoring with no user-facing change).
    • No QE (e.g. trivial change with high reviewer's confidence, or per agreement with the QE team).

@openshift-ci OpenShift CI
Copy link

openshift-ci bot commented Nov 27, 2023

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from jotak. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@jotak jotak added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Feb 16, 2024
@github-actions GitHub Actions
Copy link

New images:

  • quay.io/netobserv/network-observability-operator:ea0629c
  • quay.io/netobserv/network-observability-operator-bundle:v0.0.0-ea0629c
  • quay.io/netobserv/network-observability-operator-catalog:v0.0.0-ea0629c

They will expire after two weeks.

To deploy this build:

# Direct deployment, from operator repo
IMAGE=quay.io/netobserv/network-observability-operator:ea0629c make deploy

# Or using operator-sdk
operator-sdk run bundle quay.io/netobserv/network-observability-operator-bundle:v0.0.0-ea0629c

Or as a Catalog Source:

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: netobserv-dev
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/netobserv/network-observability-operator-catalog:v0.0.0-ea0629c
  displayName: NetObserv development catalog
  publisher: Me
  updateStrategy:
    registryPoll:
      interval: 1m

@github-actions github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Feb 16, 2024
@jotak jotak added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Feb 16, 2024
@jotak
Copy link
Member Author

jotak commented Feb 16, 2024

@msherif1234 ok I finally have all working with mounted certificates etc.
There's also a new agent image: netobserv/netobserv-ebpf-agent#268

@github-actions GitHub Actions
Copy link

New images:

  • quay.io/netobserv/network-observability-operator:f931b70
  • quay.io/netobserv/network-observability-operator-bundle:v0.0.0-f931b70
  • quay.io/netobserv/network-observability-operator-catalog:v0.0.0-f931b70

They will expire after two weeks.

To deploy this build:

# Direct deployment, from operator repo
IMAGE=quay.io/netobserv/network-observability-operator:f931b70 make deploy

# Or using operator-sdk
operator-sdk run bundle quay.io/netobserv/network-observability-operator-bundle:v0.0.0-f931b70

Or as a Catalog Source:

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: netobserv-dev
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/netobserv/network-observability-operator-catalog:v0.0.0-f931b70
  displayName: NetObserv development catalog
  publisher: Me
  updateStrategy:
    registryPoll:
      interval: 1m

@github-actions github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Feb 26, 2024
@jotak jotak added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Feb 26, 2024
@github-actions GitHub Actions
Copy link

New images:

  • quay.io/netobserv/network-observability-operator:baa600b
  • quay.io/netobserv/network-observability-operator-bundle:v0.0.0-baa600b
  • quay.io/netobserv/network-observability-operator-catalog:v0.0.0-baa600b

They will expire after two weeks.

To deploy this build:

# Direct deployment, from operator repo
IMAGE=quay.io/netobserv/network-observability-operator:baa600b make deploy

# Or using operator-sdk
operator-sdk run bundle quay.io/netobserv/network-observability-operator-bundle:v0.0.0-baa600b

Or as a Catalog Source:

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: netobserv-dev
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/netobserv/network-observability-operator-catalog:v0.0.0-baa600b
  displayName: NetObserv development catalog
  publisher: Me
  updateStrategy:
    registryPoll:
      interval: 1m

@jotak jotak changed the title NETOBSERV-627 merge Agent and FLP (PoC) NETOBSERV-627: merge Agent and FLP Feb 28, 2024
@openshift-ci-robot
Copy link
Collaborator

openshift-ci-robot commented Feb 28, 2024 •
edited by openshift-ci bot
Loading

@jotak: This pull request references NETOBSERV-627 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.16.0" version, but no target version was set.

In response to this:

Description

Do not deploy FLP: instead, configure the Agent to start FLP in-process. This only works without Kafka.

This is a PoC: consider merging only if it improves performances.

Dependencies

n/a

Checklist

If you are not familiar with our processes or don't know what to answer in the list below, let us know in a comment: the maintainers will take care of that.

  • Is this PR backed with a JIRA ticket? If so, make sure it is written as a title prefix (in general, PRs affecting the NetObserv/Network Observability product should be backed with a JIRA ticket - especially if they bring user facing changes).
  • Does this PR require product documentation?
  • If so, make sure the JIRA epic is labelled with "documentation" and provides a description relevant for doc writers, such as use cases or scenarios. Any required step to activate or configure the feature should be documented there, such as new CRD knobs.
  • Does this PR require a product release notes entry?
  • If so, fill in "Release Note Text" in the JIRA.
  • Is there anything else the QE team should know before testing? E.g: configuration changes, environment setup, etc.
  • If so, make sure it is described in the JIRA ticket.
  • QE requirements (check 1 from the list):
  • Standard QE validation, with pre-merge tests unless stated otherwise.
  • Regression tests only (e.g. refactoring with no user-facing change).
  • No QE (e.g. trivial change with high reviewer's confidence, or per agreement with the QE team).

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Collaborator

openshift-ci-robot commented Feb 28, 2024 •
edited by openshift-ci bot
Loading

@jotak: This pull request references NETOBSERV-627 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.16.0" version, but no target version was set.

In response to this:

Description

Do not deploy FLP: instead, configure the Agent to start FLP in-process. This only works without Kafka.

This is a PoC: consider merging only if it improves performances.

Dependencies

n/a

Checklist

If you are not familiar with our processes or don't know what to answer in the list below, let us know in a comment: the maintainers will take care of that.

  • Is this PR backed with a JIRA ticket? If so, make sure it is written as a title prefix (in general, PRs affecting the NetObserv/Network Observability product should be backed with a JIRA ticket - especially if they bring user facing changes).
  • Does this PR require product documentation?
  • If so, make sure the JIRA epic is labelled with "documentation" and provides a description relevant for doc writers, such as use cases or scenarios. Any required step to activate or configure the feature should be documented there, such as new CRD knobs.
  • Does this PR require a product release notes entry?
  • If so, fill in "Release Note Text" in the JIRA.
  • Is there anything else the QE team should know before testing? E.g: configuration changes, environment setup, etc.
  • If so, make sure it is described in the JIRA ticket.
  • QE requirements (check 1 from the list):
  • Standard QE validation, with pre-merge tests unless stated otherwise.
  • Regression tests only (e.g. refactoring with no user-facing change).
  • No QE (e.g. trivial change with high reviewer's confidence, or per agreement with the QE team).

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Collaborator

openshift-ci-robot commented Feb 28, 2024 •
edited by openshift-ci bot
Loading

@jotak: This pull request references NETOBSERV-627 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.16.0" version, but no target version was set.

In response to this:

Description

Do not deploy FLP: instead, configure the Agent to start FLP in-process. This only works without Kafka.

This is a PoC: consider merging only if it improves performances.

Dependencies

n/a

Checklist

If you are not familiar with our processes or don't know what to answer in the list below, let us know in a comment: the maintainers will take care of that.

  • Is this PR backed with a JIRA ticket? If so, make sure it is written as a title prefix (in general, PRs affecting the NetObserv/Network Observability product should be backed with a JIRA ticket - especially if they bring user facing changes).
  • Does this PR require product documentation?
  • If so, make sure the JIRA epic is labelled with "documentation" and provides a description relevant for doc writers, such as use cases or scenarios. Any required step to activate or configure the feature should be documented there, such as new CRD knobs.
  • Does this PR require a product release notes entry?
  • If so, fill in "Release Note Text" in the JIRA.
  • Is there anything else the QE team should know before testing? E.g: configuration changes, environment setup, etc.
  • If so, make sure it is described in the JIRA ticket.
  • QE requirements (check 1 from the list):
  • Standard QE validation, with pre-merge tests unless stated otherwise.
  • Regression tests only (e.g. refactoring with no user-facing change).
  • No QE (e.g. trivial change with high reviewer's confidence, or per agreement with the QE team).

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@jotak jotak marked this pull request as ready for review February 28, 2024 17:52
@openshift-ci-robot
Copy link
Collaborator

openshift-ci-robot commented Mar 6, 2024 •
edited by openshift-ci bot
Loading

@jotak: This pull request references NETOBSERV-627 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.16.0" version, but no target version was set.

In response to this:

Description

Do not deploy FLP: instead, configure the Agent to start FLP in-process. This only works without Kafka.

This is a PoC: consider merging only if it improves performances. It does

Dependencies

n/a

Checklist

If you are not familiar with our processes or don't know what to answer in the list below, let us know in a comment: the maintainers will take care of that.

  • Is this PR backed with a JIRA ticket? If so, make sure it is written as a title prefix (in general, PRs affecting the NetObserv/Network Observability product should be backed with a JIRA ticket - especially if they bring user facing changes).
  • Does this PR require product documentation?
  • If so, make sure the JIRA epic is labelled with "documentation" and provides a description relevant for doc writers, such as use cases or scenarios. Any required step to activate or configure the feature should be documented there, such as new CRD knobs.
  • Does this PR require a product release notes entry?
  • If so, fill in "Release Note Text" in the JIRA.
  • Is there anything else the QE team should know before testing? E.g: configuration changes, environment setup, etc.
  • If so, make sure it is described in the JIRA ticket.
  • QE requirements (check 1 from the list):
  • Standard QE validation, with pre-merge tests unless stated otherwise.
  • Regression tests only (e.g. refactoring with no user-facing change).
  • No QE (e.g. trivial change with high reviewer's confidence, or per agreement with the QE team).

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@github-actions github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Mar 6, 2024
@jotak jotak added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Mar 6, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Mar 6, 2024

New images:

  • quay.io/netobserv/network-observability-operator:c62a69d
  • quay.io/netobserv/network-observability-operator-bundle:v0.0.0-c62a69d
  • quay.io/netobserv/network-observability-operator-catalog:v0.0.0-c62a69d

They will expire after two weeks.

To deploy this build:

# Direct deployment, from operator repo
IMAGE=quay.io/netobserv/network-observability-operator:c62a69d make deploy

# Or using operator-sdk
operator-sdk run bundle quay.io/netobserv/network-observability-operator-bundle:v0.0.0-c62a69d

Or as a Catalog Source:

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: netobserv-dev
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/netobserv/network-observability-operator-catalog:v0.0.0-c62a69d
  displayName: NetObserv development catalog
  publisher: Me
  updateStrategy:
    registryPoll:
      interval: 1m

@jotak jotak mentioned this pull request Mar 6, 2024
Merged
jpinsonneau
jpinsonneau reviewed Mar 6, 2024
View reviewed changes
controllers/ebpf/agent_controller.go
@@ -64,7 +66,7 @@ const (

const (
exportKafka = "kafka"
exportGRPC = "grpc"
exportFLP = "direct-flp"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the plan to force users not using kafka to have direct-flp here ?

I wonder if in some cases we want them separated:

  • For security pusposes for example when using ebpf privileged mode
  • For resources consumption repartition if the observability stack must have a very low impact on a particular node
  • For small clusters where a single FLP could be enough

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd say removing the non-encrypted grpc connection will always be more secure than having it?
I'm not sure to understand the two other points, as having merged agent+FLP is more lightweight than having both of them as two different pods (they're both daemonsets / one per node)

@jotak
NETOBSERV-627 merge Agent and FLP
ae3b357
@jotak
NETOBSERV-627: direct-flp
e743bbe 
- move all related code in file in_process_reconciler.go
- reconcile Loki roles, prometheus service/sm, cert watching etc.
@jotak
Refactor to remove old reconcilers + tests refactoring
5881bc4 
- Removing the "subReconcilers" in FLP controller, as FLP is now only
  deployed with Kafka ...
- ... but still keep a disctinct pseudo-reconciler "in-process", which
  has FLP logic but is triggered from Agent controller
- Big tests refactoring, started due to the reconciler removal and went
  further by factorizing some patterns like checking that a list of
objects are deleted/created/owned/etc.
@github-actions github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Mar 11, 2024
@openshift-ci OpenShift CI
Copy link

openshift-ci bot commented Mar 11, 2024

@jotak: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-operator 47297da link false /test e2e-operator

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jpinsonneau jpinsonneau jpinsonneau left review comments

@msherif1234 msherif1234 Awaiting requested review from msherif1234

Assignees
No one assigned
Labels
jira/valid-reference
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@jotak @openshift-ci-robot @jpinsonneau @openshift-merge-robot