adding ebpf collector

[povilasv]

We had a PR open #833 to add ebpf collection to opentelemetry-collector chart

Let's discuss it.

There seems to be three components suggested:

• opentelemetry-ebpf-cloud-collector
• opentelemetry-ebpf-reducer
• k8sCollector, which has two images:

1. opentelemetry-ebpf-k8s-watcher
2. opentelemetry-ebpf-k8s-relay

I think we mostly we want this as a separate helm chart. I'm not sure how everything relates and works. Should everything be a single chart? How do we propose new charts and who should maintain it?

cc: @TylerHelmuth @dmitryax @atoulme @nicolastakashi

[nicolastakashi]

Thanks for opening this issue @povilasv 😍
For historical context, the eBPF collector has donated a few months ago to the OpenTelemetry initiative, since then people are working to improve this collector is easy to run and operate using the OTEL standards, as part of this job there's an issue with the project repository asking to have this collector as part as the OTEL helm charts, as described here.

To better understand, below you can find an image explaining the collector architecture I got from the Splunk helm chart, available here.

With that in mind, I would say that all the listed components should be available in only one helm chart.

Regarding the Chart Maintenance, I'm more than available to work on this, if everybody is Ok with that.

[atoulme]

This chart can be deployed independently from a collector chart. It indeed contains 3 components that should be deployed together.

[TylerHelmuth]

This chart can be deployed independently from a collector chart. It indeed contains 3 components that should be deployed together.

Based on this I definitely think it should be its own chart.

[samiura]

➕ 1 ☝🏽

[jvoravong]

➕ 1 ☝🏽

[nicolastakashi]

Look we have an agreement!
@TylerHelmuth @povilasv may I change the PR and add a new chart?

[TylerHelmuth]

@nicolastakashi before we review the PR a few more questions:

1. Can you layout in detail what the chart will install
2. Is this chart testable with chart-testing?
3. How stable is the OTel eBPF agent(s)? Will the chart need constant support as the agent(s) change?
4. Who from the OTel eBPF SIG is willing to be a code owner for this chart?

[nicolastakashi]

Sure!

1. Can you layout in detail what the chart will install

As we can see on the current eBPF collector architecture, this collector is composed of 3 main components:

Kernel Collector

It's the brain of this solution, this component is responsible for attaching eBPF probs and collecting (at this moment), network information at the Kernel Level and pushing the collected information into the next component named Reducer.

For more deep details about the Kernel Collector component, I do recommend checking this doc available on the project repository.

Reducer (Required)

The role of the reducer is to combine information received from eBPF collectors into metrics. Metrics are aggregated across time and dimensions. The output is in the form of timestamped data points, which can be stored in a time-series database.

For more deep details about the Reducer component, I do recommend checking this doc available on the project repository.

Kubernetes Collector

The Kubernetes collector gathers information from a Kubernetes API server on events like pod creation and deletion and pushes that information to the Reducer.

The information collected by the Kubernetes Collector will be used to enrich the information collected by the Kernel Collector adding information such as source and destination. This is an optional component, but we do recommend users use it to have better visibility.

The project maintainers aim to simplify this component since it's composed of two different binaries written in two different languages C and Go, as you can see here.

So this is the component that I see in the future having some changes (maybe breaking), but please @atoulme give me your view about that.

For more deep details about the Kubernetes Collector component, I do recommend checking this doc available on the project repository.

Cloud Collector

The cloud collector gathers metadata from a cloud provider and pushes that information to the Reducer.
With that collector, we can enrich the information collected by the Kernel Collector with some information such as availability zones.

Currently, there's only support for AWS and GCP

For more deep details about the Cloud Collector component, I do recommend checking this doc available on the project repository.

---

2. Is this chart testable with chart-testing?

From my understanding yes, I can't see a reason why this is not possible, wdyt @atoulme?

---

3. How stable is the OTel eBPF agent(s)? Will the chart need constant support as the agent(s) change?

In general, I would say stable, most of the things are working on top of the existing components so probably only changing the image version should be required.

The only component that I can see a possible breaking change in the future is the Kubernetes Collector.

Of course, the project may evolve in the future and new components appear.

---

4. Who from the OTel eBPF SIG is willing to be a code owner for this chart?

@atoulme I need your help with that.

I'm not part of OTel eBPF Sig but I'm available to help with it.

---

@TylerHelmuth let me know if the description is good 😄

[TylerHelmuth]

@nicolastakashi thank you for those details.

@atoulme we'll need an answer to Who from the OTel eBPF SIG is willing to be a code owner for this chart? before we can proceed.

Also, as a side note, using the word Collector in relation to an OTel component but not referring to the OpenTelemetry Collector is pretty confusing, has the OTel eBPF SIG considered changing terms?

[atoulme]

I can volunteer to be a code owner of this chart.

No, we have not worked out our naming. I will open an issue for this.

[TylerHelmuth]

I think this chart meets our requirements. @open-telemetry/helm-approvers do you agree?

[dmitryax]

I support it

[povilasv]

I support

[nicolastakashi]

Hey folks!
Are we good to move forward with this?

[TylerHelmuth]

Yes

[nicolastakashi]

Hey folks @TylerHelmuth @atoulme it's ready for review: #855

Just need to write the readme.md but it's fine for a first look

[TylerHelmuth]

@nicolastakashi thanks for working on this. I will slowly take a look over the next week.

[JaredTan95]

This issue can be closed? as #855 finished.

@nicolastakashi @open-telemetry/helm-maintainers