Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the ability to set tail_from_end when gathering logs from volumes #984

Merged
merged 1 commit into from
Dec 10, 2024
Merged

Add the ability to set tail_from_end when gathering logs from volumes #984

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/platform-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ on:
- 'charts/**'
- '!charts/k8s-monitoring-v1/**'
pull_request:
types: [ labeled ]
types: [labeled]
paths:
- 'charts/**'
- '!charts/k8s-monitoring-v1/**'
Expand Down
36 changes: 18 additions & 18 deletions charts/feature-cluster-metrics/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,30 +17,30 @@ not on the list will be dropped. With [metrics tuning](#metrics-tuning--allow-li

This chart includes the ability to collect metrics from the following:

* The Kubernetes cluster itself
* Sources like the Kubelet and cAdvisor
* Common supporting services like [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) and
[Node Exporter](https://github.com/prometheus/node_exporter)
* Systems to capture additional data like Kepler
* The Kubernetes cluster itself
* Sources like the Kubelet and cAdvisor
* Common supporting services like [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) and
[Node Exporter](https://github.com/prometheus/node_exporter)
* Systems to capture additional data like Kepler

### Metrics sources

The Cluster Metrics feature of the Kubernetes Observability Helm chart includes the following metric systems and
their default allow lists:

| Metric source | Gathers information about | Allow list |
| --- | --- | --- |
| API Server | Kubernetes API Server | NA |
| [cAdvisor](https://github.com/google/cadvisor)| Containers on each node | [default-allow-lists/cadvisor.yaml](./default-allow-lists/cadvisor.yaml) |
| [Kepler](https://sustainable-computing.io/) | Kubernetes cluster | [default-allow-lists/kepler.yaml](./default-allow-lists/kepler.yaml) |
| Kube Controller Manager | Kubernetes Controller Manager | NA |
| Kube Proxy | Kube Proxy | NA|
| Kube Scheduler | Kube Scheduler | NA|
| Kubelet | Kubernetes information on each node | [default-allow-lists/kubelet.yaml](./default-allow-lists/kubelet.yaml) |
| [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) |Kubernetes
resources inside the cluster | [default-allow-lists/kube-state-metrics.yaml](./default-allow-lists/kube-state-metrics.yaml) |
| [Node Exporter](https://github.com/prometheus/node_exporter) | Linux Kubernetes nodes | [default-allow-lists/node-exporter.yaml](./default-allow-lists/node-exporter.yaml), [default-allow-lists/node-exporter-integration.yaml](./default-allow-lists/node-exporter-integration.yaml) |
| [Windows Exporter](https://github.com/prometheus-community/windows_exporter) | Windows Kubernetes nodes | [default-allow-lists/windows-exporter.yaml](./default-allow-lists/windows-exporter.yaml) |
| Metric source | Gathers information about | Allow list |
|------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| API Server | Kubernetes API Server | NA |
| [cAdvisor](https://github.com/google/cadvisor) | Containers on each node | [default-allow-lists/cadvisor.yaml](./default-allow-lists/cadvisor.yaml) |
| [Kepler](https://sustainable-computing.io/) | Kubernetes cluster | [default-allow-lists/kepler.yaml](./default-allow-lists/kepler.yaml) |
| Kube Controller Manager | Kubernetes Controller Manager | NA |
| Kube Proxy | Kube Proxy | NA |
| Kube Scheduler | Kube Scheduler | NA |
| Kubelet | Kubernetes information on each node | [default-allow-lists/kubelet.yaml](./default-allow-lists/kubelet.yaml) |
| [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) | Kubernetes | |
| resources inside the cluster | [default-allow-lists/kube-state-metrics.yaml](./default-allow-lists/kube-state-metrics.yaml) | |
| [Node Exporter](https://github.com/prometheus/node_exporter) | Linux Kubernetes nodes | [default-allow-lists/node-exporter.yaml](./default-allow-lists/node-exporter.yaml), [default-allow-lists/node-exporter-integration.yaml](./default-allow-lists/node-exporter-integration.yaml) |
| [Windows Exporter](https://github.com/prometheus-community/windows_exporter) | Windows Kubernetes nodes | [default-allow-lists/windows-exporter.yaml](./default-allow-lists/windows-exporter.yaml) |

## Metrics tuning and allow lists

Expand Down
36 changes: 18 additions & 18 deletions charts/feature-cluster-metrics/README.md.gotmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,30 +20,30 @@ not on the list will be dropped. With [metrics tuning](#metrics-tuning--allow-li

This chart includes the ability to collect metrics from the following:

* The Kubernetes cluster itself
* Sources like the Kubelet and cAdvisor
* Common supporting services like [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) and
[Node Exporter](https://github.com/prometheus/node_exporter)
* Systems to capture additional data like Kepler
* The Kubernetes cluster itself
* Sources like the Kubelet and cAdvisor
* Common supporting services like [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) and
[Node Exporter](https://github.com/prometheus/node_exporter)
* Systems to capture additional data like Kepler

### Metrics sources

The Cluster Metrics feature of the Kubernetes Observability Helm chart includes the following metric systems and
their default allow lists:

| Metric source | Gathers information about | Allow list |
| --- | --- | --- |
| API Server | Kubernetes API Server | NA |
| [cAdvisor](https://github.com/google/cadvisor)| Containers on each node | [default-allow-lists/cadvisor.yaml](./default-allow-lists/cadvisor.yaml) |
| [Kepler](https://sustainable-computing.io/) | Kubernetes cluster | [default-allow-lists/kepler.yaml](./default-allow-lists/kepler.yaml) |
| Kube Controller Manager | Kubernetes Controller Manager | NA |
| Kube Proxy | Kube Proxy | NA|
| Kube Scheduler | Kube Scheduler | NA|
| Kubelet | Kubernetes information on each node | [default-allow-lists/kubelet.yaml](./default-allow-lists/kubelet.yaml) |
| [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) |Kubernetes
resources inside the cluster | [default-allow-lists/kube-state-metrics.yaml](./default-allow-lists/kube-state-metrics.yaml) |
| [Node Exporter](https://github.com/prometheus/node_exporter) | Linux Kubernetes nodes | [default-allow-lists/node-exporter.yaml](./default-allow-lists/node-exporter.yaml), [default-allow-lists/node-exporter-integration.yaml](./default-allow-lists/node-exporter-integration.yaml) |
| [Windows Exporter](https://github.com/prometheus-community/windows_exporter) | Windows Kubernetes nodes | [default-allow-lists/windows-exporter.yaml](./default-allow-lists/windows-exporter.yaml) |
| Metric source | Gathers information about | Allow list |
|------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| API Server | Kubernetes API Server | NA |
| [cAdvisor](https://github.com/google/cadvisor) | Containers on each node | [default-allow-lists/cadvisor.yaml](./default-allow-lists/cadvisor.yaml) |
| [Kepler](https://sustainable-computing.io/) | Kubernetes cluster | [default-allow-lists/kepler.yaml](./default-allow-lists/kepler.yaml) |
| Kube Controller Manager | Kubernetes Controller Manager | NA |
| Kube Proxy | Kube Proxy | NA |
| Kube Scheduler | Kube Scheduler | NA |
| Kubelet | Kubernetes information on each node | [default-allow-lists/kubelet.yaml](./default-allow-lists/kubelet.yaml) |
| [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) | Kubernetes | |
| resources inside the cluster | [default-allow-lists/kube-state-metrics.yaml](./default-allow-lists/kube-state-metrics.yaml) | |
| [Node Exporter](https://github.com/prometheus/node_exporter) | Linux Kubernetes nodes | [default-allow-lists/node-exporter.yaml](./default-allow-lists/node-exporter.yaml), [default-allow-lists/node-exporter-integration.yaml](./default-allow-lists/node-exporter-integration.yaml) |
| [Windows Exporter](https://github.com/prometheus-community/windows_exporter) | Windows Kubernetes nodes | [default-allow-lists/windows-exporter.yaml](./default-allow-lists/windows-exporter.yaml) |

## Metrics tuning and allow lists

Expand Down
6 changes: 6 additions & 0 deletions charts/feature-pod-logs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,3 +62,9 @@ Be sure perform actual integration testing in a live environment in the main [k8
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| global.platform | string | `""` | The specific platform for this cluster. Will enable compatibility for some platforms. Supported options: (empty) or "openshift". |

### Volume Log Gathering

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| volumeGatherSettings.onlyGatherNewLogLines | bool | `false` | Only gather new log lines since this was deployed. Do not gather historical log lines. |
3 changes: 3 additions & 0 deletions charts/feature-pod-logs/templates/_volumes.alloy.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ local.file_match "pod_logs" {

loki.source.file "pod_logs" {
targets = local.file_match.pod_logs.targets
{{- if .Values.volumeGatherSettings.onlyGatherNewLogLines }}
tail_from_end = {{ .Values.volumeGatherSettings.onlyGatherNewLogLines }}
{{- end }}
forward_to = [loki.process.pod_logs.receiver]
}
{{- end -}}
8 changes: 8 additions & 0 deletions charts/feature-pod-logs/values.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,14 @@
},
"namespaces": {
"type": "array"
},
"volumeGatherSettings": {
"type": "object",
"properties": {
"onlyGatherNewLogLines": {
"type": "boolean"
}
}
}
}
}
6 changes: 6 additions & 0 deletions charts/feature-pod-logs/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,12 @@ labels:
annotations:
job: k8s.grafana.com/logs.job

# Settings specific for gathering Pod logs using the "volumes" gather method.
volumeGatherSettings:
# -- Only gather new log lines since this was deployed. Do not gather historical log lines.
# @section -- Volume Log Gathering
onlyGatherNewLogLines: false

# -- Stage blocks to be added to the loki.process component for pod logs.
# ([docs](https://grafana.com/docs/alloy/latest/reference/components/loki/loki.process/#blocks))
# This value is templated so that you can refer to other values from this file.
Expand Down
1 change: 1 addition & 0 deletions charts/k8s-monitoring-v1/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -481,6 +481,7 @@ The Prometheus and Loki services may be hosted on the same cluster, or remotely
| logs.pod_logs.labels | object | `{"app_kubernetes_io_name":"app.kubernetes.io/name"}` | Loki labels to set with values copied from the Kubernetes Pod labels. Format: `<loki_label>: <kubernetes_label>`. |
| logs.pod_logs.namespaces | list | `[]` | Only capture logs from pods in these namespaces (`[]` means all namespaces). |
| logs.pod_logs.structuredMetadata | object | `{}` | List of labels to turn into structured metadata. If your Loki instance does not support structured metadata, leave this empty. Format: `<structured metadata>: <Loki label>`. |
| logs.pod_logs.volumeGatherSettings.onlyGatherNewLogLines | bool | `false` | Only gather new log lines since this was deployed. Do not gather historical log lines. |

### Logs Receiver

Expand Down
3 changes: 3 additions & 0 deletions charts/k8s-monitoring-v1/templates/alloy_config/_pod_logs_discovery.alloy.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,9 @@ local.file_match "pod_logs" {

loki.source.file "pod_logs" {
targets = local.file_match.pod_logs.targets
{{- if .Values.logs.pod_logs.volumeGatherSettings.onlyGatherNewLogLines }}
tail_from_end = {{ .Values.logs.pod_logs.volumeGatherSettings.onlyGatherNewLogLines }}
{{- end }}
forward_to = [loki.process.pod_logs.receiver]
}
{{ else if eq .Values.logs.pod_logs.gatherMethod "api" }}
Expand Down
8 changes: 8 additions & 0 deletions charts/k8s-monitoring-v1/values.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1167,6 +1167,14 @@
},
"structuredMetadata": {
"type": "object"
},
"volumeGatherSettings": {
"type": "object",
"properties": {
"onlyGatherNewLogLines": {
"type": "boolean"
}
}
}
}
},
Expand Down
6 changes: 6 additions & 0 deletions charts/k8s-monitoring-v1/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1707,6 +1707,12 @@ logs:
# @section -- Logs Scrape: Pod Logs
structuredMetadata: {}

# Settings specific for gathering Pod logs using the "volumes" gather method.
volumeGatherSettings:
# -- Only gather new log lines since this was deployed. Do not gather historical log lines.
# @section -- Logs Scrape: Pod Logs
onlyGatherNewLogLines: false

# PodLog Objects
podLogsObjects:
# -- Enable discovery of Grafana Alloy PodLogs objects.
Expand Down
2 changes: 1 addition & 1 deletion charts/k8s-monitoring/Chart.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,4 +45,4 @@ dependencies:
repository: https://grafana.github.io/helm-charts
version: 0.10.1
digest: sha256:e2eab36a3b814d7ec09942c3fe46a5c76d606f7c3022628905c41fba7a47b405
generated: "2024-12-03T20:08:14.380789162Z"
generated: "2024-12-09T19:06:04.597148-06:00"
Binary file modified charts/k8s-monitoring/charts/k8s-monitoring-feature-annotation-autodiscovery-1.0.0.tgz
View file
Open in desktop
Binary file not shown.
Binary file modified charts/k8s-monitoring/charts/k8s-monitoring-feature-application-observability-1.0.0.tgz
View file
Open in desktop
Binary file not shown.
Binary file modified charts/k8s-monitoring/charts/k8s-monitoring-feature-auto-instrumentation-1.0.0.tgz
View file
Open in desktop
Binary file not shown.
Binary file modified charts/k8s-monitoring/charts/k8s-monitoring-feature-cluster-events-1.0.0.tgz
View file
Open in desktop
Binary file not shown.
Binary file modified charts/k8s-monitoring/charts/k8s-monitoring-feature-cluster-metrics-1.0.0.tgz
View file
Open in desktop
Binary file not shown.
Binary file modified charts/k8s-monitoring/charts/k8s-monitoring-feature-integrations-1.0.0.tgz
View file
Open in desktop
Binary file not shown.
Binary file modified charts/k8s-monitoring/charts/k8s-monitoring-feature-node-logs-1.0.0.tgz
View file
Open in desktop
Binary file not shown.
Binary file modified charts/k8s-monitoring/charts/k8s-monitoring-feature-pod-logs-1.0.0.tgz
View file
Open in desktop
Binary file not shown.
Binary file modified charts/k8s-monitoring/charts/k8s-monitoring-feature-profiling-1.0.0.tgz
View file
Open in desktop
Binary file not shown.
Binary file modified charts/k8s-monitoring/charts/k8s-monitoring-feature-prometheus-operator-objects-1.0.0.tgz
View file
Open in desktop
Binary file not shown.
2 changes: 1 addition & 1 deletion charts/k8s-monitoring/tests/integration/auth/configs/nginx-auth-gateway.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ serverBlock: |-

proxy_pass http://loki.loki.svc:3100/;
}

location /metrics/bearer/ {
if ($http_authorization != "Bearer my-bearer-token") {
return 401;
Expand Down
Loading