[PROCS-3710] Add ability to run Processes features on Core Agent #1121

daniel-taf · 2024-03-14T21:46:07Z

What does this PR do?

This PR adds the ability to run all Processes features on the core agent. A config type, ProcessesRunInCoreAgent, is added to all Processes feature types to facilitate this. Because each feature can have its own value set, the source of truth is handled with the following precedence: Live Processes > Live Containers > Process Discovery.

Motivation

What inspired you to submit this pull request?

Additional Notes

This is meant to be an experimental feature to be run internally for now.
Depends on [PROCS-3915] Add agent version check to Orchestrator Explorer feature to determine if process agent is required #1125 to avoid crash loops when turned on
This does require agent version 7.52.0 to run, so a version check should be added as a follow up.

Minimum Agent Versions

Are there minimum versions of the Datadog Agent and/or Cluster Agent required?

Agent: v7.52.0
Cluster Agent: N/A

Describe your test plan

Each feature was tested in both configurations (core agent and process agent) on a local kind cluster.
Live Processes Example:
Process Agent

  features:
    liveProcessCollection:
      enabled: true

Results in:

❯ kubectl get pod datadog-agent-9jvw9 -o json | jq '.spec.containers[].name'
"agent"
"trace-agent"
"process-agent"
❯ kubectl logs datadog-agent-9jvw9 --container process-agent | grep checks
2024-03-21 19:28:00 UTC | PROCESS | INFO | (pkg/process/runner/runner.go:286 in Run) | Starting process-agent with enabled checks=[process rtprocess]

Core Agent

  features:
    liveProcessCollection:
      enabled: true
      runInCoreAgent:
        enabled: true

Results in:

❯ kubectl get pod datadog-agent-4hr8f -o json | jq '.spec.containers[].name'
"agent"
"trace-agent"
❯ kubectl logs datadog-agent-4hr8f --container agent | grep checks
2024-03-21 19:56:09 UTC | CORE | INFO | (pkg/process/runner/runner.go:286 in Run) | Starting process-agent with enabled checks=[process rtprocess]

Checklist

PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
PR has a milestone or the qa/skip-qa label

github-actions

This pull request does not contain a valid label. Please add one of the following labels: bug, enhancement, refactoring, documentation, tooling, dependencies

codecov-commenter · 2024-03-20T19:44:45Z

Codecov Report

Attention: Patch coverage is 93.79310% with 9 lines in your changes are missing coverage. Please review.

Project coverage is 59.21%. Comparing base (6571686) to head (2f858dd).

Additional details and impacted files

@@                                    Coverage Diff                                     @@
##           daniel.tafoya/orchestrator-process-agent-container-fix    #1121      +/-   ##
==========================================================================================
+ Coverage                                                   58.98%   59.21%   +0.22%     
==========================================================================================
  Files                                                         174      174              
  Lines                                                       21387    21511     +124     
==========================================================================================
+ Hits                                                        12616    12737     +121     
- Misses                                                       8004     8006       +2     
- Partials                                                      767      768       +1

Flag	Coverage Δ
unittests	`59.21% <93.79%> (+0.22%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
apis/datadoghq/v2alpha1/datadogagent_types.go	`100.00% <ø> (ø)`
...ollers/datadogagent/feature/liveprocess/feature.go	`93.27% <100.00%> (+6.69%)`	⬆️
...lers/datadogagent/feature/livecontainer/feature.go	`88.23% <90.69%> (+0.35%)`	⬆️
...s/datadogagent/feature/processdiscovery/feature.go	`85.00% <91.52%> (+11.53%)`	⬆️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6571686...2f858dd. Read the comment docs.

daniel-taf · 2024-03-21T21:48:57Z

controllers/datadogagent/feature/liveprocess/feature.go

+	containerName := apicommonv1.ProcessAgentContainerName
+	runInCoreAgent := f.runInCoreAgentCfg != nil && f.runInCoreAgentCfg.enabled
+	if runInCoreAgent {
+		containerName = apicommonv1.CoreAgentContainerName
+	}
+	runInCoreAgentEnvVar := &corev1.EnvVar{
+		Name:  apicommon.DDProcessConfigRunInCoreAgent,
+		Value: apiutils.BoolToString(&runInCoreAgent),
+	}
+	managers.EnvVar().AddEnvVarToContainer(apicommonv1.ProcessAgentContainerName, runInCoreAgentEnvVar)
+	managers.EnvVar().AddEnvVarToContainer(apicommonv1.CoreAgentContainerName, runInCoreAgentEnvVar)


Any good place to put util functions that are shared between certain features? Lot of shared code could be reduced.

daniel-taf · 2024-03-21T22:02:13Z

apis/datadoghq/v2alpha1/datadogagent_types.go

+	// RunInCoreAgent controls enablement of running the feature in the core agent.
+	// +optional
+	RunInCoreAgent *ProcessesRunInCoreAgent `json:"runInCoreAgent,omitempty"`


It may be better to have one top level configuration for this to avoid having to handle inconsistent values as it adds a lot of logic. Is there a good way to do that? It seemed to be against the standard convention to do so.

hestonhoffman

Couple small docs edits

hestonhoffman · 2024-03-22T16:17:46Z

docs/configuration.v2alpha1.md

@@ -96,7 +96,9 @@ spec:
 | features.kubeStateMetricsCore.conf.configMap.name | Name is the name of the ConfigMap. |
 | features.kubeStateMetricsCore.enabled | Enabled enables Kube State Metrics Core. Default: true |
 | features.liveContainerCollection.enabled | Enables container collection for the Live Container View. Default: true |
+| features.liveContainerCollection.runInCoreAgent.enabled | Enabled enables running parent feature in core agent (experimental) Default: false |


Suggested change

| features.liveContainerCollection.runInCoreAgent.enabled | Enabled enables running parent feature in core agent (experimental) Default: false |

| features.liveContainerCollection.runInCoreAgent.enabled | Enabled enables running parent feature in core Agent (experimental). Default: false |

hestonhoffman · 2024-03-22T16:17:58Z

docs/configuration.v2alpha1.md

 | features.liveProcessCollection.enabled | Enabled enables Process monitoring. Default: false |
+| features.liveProcessCollection.runInCoreAgent.enabled | Enabled enables running parent feature in core agent (experimental) Default: false |


Suggested change

| features.liveProcessCollection.runInCoreAgent.enabled | Enabled enables running parent feature in core agent (experimental) Default: false |

| features.liveProcessCollection.runInCoreAgent.enabled | Enabled enables running parent feature in core Agent (experimental). Default: false |

hestonhoffman · 2024-03-22T16:18:13Z

docs/configuration.v2alpha1.md

@@ -124,6 +126,7 @@ spec:
 | features.otlp.receiver.protocols.http.enabled | Enable the OTLP/HTTP endpoint. |
 | features.otlp.receiver.protocols.http.endpoint | Endpoint for OTLP/HTTP. Default: '0.0.0.0:4318'. |
 | features.processDiscovery.enabled | Enabled enables the Process Discovery check in the Agent. Default: true |
+| features.processDiscovery.runInCoreAgent.enabled | Enabled enables running parent feature in core agent (experimental) Default: false |


Suggested change

| features.processDiscovery.runInCoreAgent.enabled | Enabled enables running parent feature in core agent (experimental) Default: false |

| features.processDiscovery.runInCoreAgent.enabled | Enabled enables running parent feature in core Agent (experimental). Default: false |

github-actions bot requested changes Mar 14, 2024

View reviewed changes

daniel-taf added the enhancement New feature or request label Mar 14, 2024

github-actions bot approved these changes Mar 14, 2024

View reviewed changes

daniel-taf added this to the v1.4.0 milestone Mar 14, 2024

daniel-taf added 4 commits March 18, 2024 13:50

Add agent version check to orchestrator explorer

6d8c1bd

Add unit tests for agent version checks

0dcdf5e

Add unit tests for agent version checks

cc302d0

fix v1 tests

e3c0869

daniel-taf changed the title ~~Add ability to run Live Processes feature on core agent~~ Add ability to ru Processes features on core agent Mar 19, 2024

daniel-taf changed the title ~~Add ability to ru Processes features on core agent~~ Add ability to run Processes features on core agent Mar 19, 2024

daniel-taf changed the base branch from main to daniel.tafoya/orchestrator-process-agent-container-fix March 19, 2024 19:53

daniel-taf changed the title ~~Add ability to run Processes features on core agent~~ Add ability to run Processes features on Core Agent Mar 19, 2024

daniel-taf added 6 commits March 19, 2024 18:09

Fix generated files

6571686

Add run in core agent to live processes feature

43e9710

Add tests

33c24fd

Rework container logic

447afdb

Add run in core agent to other processes features

7f7caa2

rework features and tests

27eb10e

daniel-taf force-pushed the daniel.tafoya/processes-run-in-core branch from fea8213 to 27eb10e Compare March 20, 2024 19:24

make generate

0a928b8

daniel-taf changed the base branch from daniel.tafoya/orchestrator-process-agent-container-fix to main March 20, 2024 19:30

daniel-taf changed the base branch from main to daniel.tafoya/orchestrator-process-agent-container-fix March 20, 2024 19:31

fix v1 test

9d52a7d

Add mounts to processes features

2a7c79b

celenechang modified the milestones: v1.4.0, v1.6.0 Mar 21, 2024

daniel-taf added 3 commits March 21, 2024 11:49

fix v1 tests

494f3ff

Add comments

61f087f

Add better handling for inconsistent values

8ed2942

update docs

2f858dd

daniel-taf changed the title ~~Add ability to run Processes features on Core Agent~~ [PROCS-3710] Add ability to run Processes features on Core Agent Mar 21, 2024

daniel-taf commented Mar 21, 2024

View reviewed changes

remove unused function

c850663

daniel-taf marked this pull request as ready for review March 21, 2024 21:59

daniel-taf requested review from a team as code owners March 21, 2024 21:59

daniel-taf commented Mar 21, 2024

View reviewed changes

hestonhoffman approved these changes Mar 22, 2024

View reviewed changes

daniel-taf force-pushed the daniel.tafoya/orchestrator-process-agent-container-fix branch from 9cf434c to 3072e4e Compare March 26, 2024 14:51

Base automatically changed from daniel.tafoya/orchestrator-process-agent-container-fix to main March 26, 2024 18:22

daniel-taf closed this Mar 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[PROCS-3710] Add ability to run Processes features on Core Agent #1121

[PROCS-3710] Add ability to run Processes features on Core Agent #1121

daniel-taf commented Mar 14, 2024 •

edited

Loading

github-actions bot left a comment

codecov-commenter commented Mar 20, 2024 •

edited

Loading

daniel-taf Mar 21, 2024 •

edited

Loading

daniel-taf Mar 21, 2024 •

edited

Loading

hestonhoffman left a comment

hestonhoffman Mar 22, 2024

hestonhoffman Mar 22, 2024

hestonhoffman Mar 22, 2024

	\| features.liveContainerCollection.runInCoreAgent.enabled \| Enabled enables running parent feature in core agent (experimental) Default: false \|
	\| features.liveContainerCollection.runInCoreAgent.enabled \| Enabled enables running parent feature in core Agent (experimental). Default: false \|

		\| features.liveProcessCollection.enabled \| Enabled enables Process monitoring. Default: false \|
		\| features.liveProcessCollection.runInCoreAgent.enabled \| Enabled enables running parent feature in core agent (experimental) Default: false \|

	\| features.processDiscovery.runInCoreAgent.enabled \| Enabled enables running parent feature in core agent (experimental) Default: false \|
	\| features.processDiscovery.runInCoreAgent.enabled \| Enabled enables running parent feature in core Agent (experimental). Default: false \|

[PROCS-3710] Add ability to run Processes features on Core Agent #1121

[PROCS-3710] Add ability to run Processes features on Core Agent #1121

Conversation

daniel-taf commented Mar 14, 2024 • edited Loading

What does this PR do?

Motivation

Additional Notes

Minimum Agent Versions

Describe your test plan

Checklist

github-actions bot left a comment

Choose a reason for hiding this comment

codecov-commenter commented Mar 20, 2024 • edited Loading

Codecov Report

daniel-taf Mar 21, 2024 • edited Loading

Choose a reason for hiding this comment

daniel-taf Mar 21, 2024 • edited Loading

Choose a reason for hiding this comment

hestonhoffman left a comment

Choose a reason for hiding this comment

hestonhoffman Mar 22, 2024

Choose a reason for hiding this comment

hestonhoffman Mar 22, 2024

Choose a reason for hiding this comment

hestonhoffman Mar 22, 2024

Choose a reason for hiding this comment

daniel-taf commented Mar 14, 2024 •

edited

Loading

codecov-commenter commented Mar 20, 2024 •

edited

Loading

daniel-taf Mar 21, 2024 •

edited

Loading

daniel-taf Mar 21, 2024 •

edited

Loading