diff --git a/operator/CHANGELOG.md b/operator/CHANGELOG.md index 171d6d945dae..97c499339b34 100644 --- a/operator/CHANGELOG.md +++ b/operator/CHANGELOG.md @@ -3,5 +3,15 @@ - [5576](https://github.com/grafana/loki/pull/5576) **xperimental**: Change endpoints for generated liveness and readiness probes - [5560](https://github.com/grafana/loki/pull/5560) **periklis**: Fix service monitor's server name for operator metrics - [5345](https://github.com/grafana/loki/pull/5345) **ronensc**: Add flag to create Prometheus rules +- [5551](https://github.com/grafana/loki/pull/5551) **sasagarw**: Document how to connect to distributor component +- [5624](https://github.com/grafana/loki/pull/5624) **periklis**: Use tenant name as id for mode openshift-logging (OpenShift) +- [5621](https://github.com/grafana/loki/pull/5621) **periklis**: Use recommended labels for LokiStack components +- [5607](https://github.com/grafana/loki/pull/5607) **periklis**: Use lokistack name as prefix for owned resources +- [5588](https://github.com/grafana/loki/pull/5588) **periklis**: Add RBAC for Prometheus service discovery to Loki component metrics (OpenShift) +- [5576](https://github.com/grafana/loki/pull/5576) **xperimental**: Change endpoints for generated liveness and readiness probes +- [5560](https://github.com/grafana/loki/pull/5560) **periklis**: Fix service monitor's server name for operator metrics +- [5345](https://github.com/grafana/loki/pull/5345) **ronensc**: Add flag to create Prometheus rules +- [4974](https://github.com/grafana/loki/pull/5432) **Red-GV**: Provide storage configuration for Azure, GCS, and Swift through common_config +- [4975](https://github.com/grafana/loki/pull/4975) **periklis**: Provide saner default for loki-operator managed chunk_target_size - [4974](https://github.com/grafana/loki/pull/5432) **Red-GV**: Provide storage configuration for Azure, GCS, and Swift through common_config - [4975](https://github.com/grafana/loki/pull/4975) **periklis**: Provide saner default for loki-operator managed chunk_target_size diff --git a/operator/docs/forwarding_logs_without_gateway.md b/operator/docs/forwarding_logs_without_gateway.md new file mode 100644 index 000000000000..cb4ab17a979e --- /dev/null +++ b/operator/docs/forwarding_logs_without_gateway.md @@ -0,0 +1,114 @@ +# Forwarding Logs to LokiStack without LokiStack Gateway + +This document describes how to send application, infrastructure, and audit logs to the Loki Distributor as different tenants using Fluentd or Vector. + +__Please read the [hacking guide](./hack_loki_operator.md) before proceeding with the following instructions.__ + +_Note:_ This document only applies to OpenShift-based deployments. + +_Disclaimer:_ This document helps to connect the forwarder (fluentd or vector) to the LokiStack by going around the authentication gateway. This is not a normal configuration for an OpenShift-based deployments and should only be used for testing if going through the gateway is no option. + +## OpenShift Logging + +[OpenShift Logging](https://github.com/openshift/cluster-logging-operator) supports [forwarding logs to an external Loki instance](https://docs.openshift.com/container-platform/4.9/logging/cluster-logging-external.html#cluster-logging-collector-log-forward-loki_cluster-logging-external) using fluentd or vector as log forwarders. +The below step-by-step guide will help you to send application, infrastructure, and audit logs to the LokiStack through the Distributor endpoint. +The steps remain same for both fluentd and vector. + +In order to enable communication between the log forwarder and the distributor, follow these steps: + +* Deploy the Loki Operator and a `lokistack` instance for [OpenShift](./hack_loki_operator.md#hacking-on-loki-operator-on-openshift). + +* Deploy the OpenShift Logging Operator from the Operator Hub or using the following command locally: + + ```console + make deploy-image deploy-catalog install + ``` + +* Create a Cluster Logging instance in the `openshift-logging` namespace with only `collection` defined. + + For fluentd: + + ```yaml + apiVersion: logging.openshift.io/v1 + kind: ClusterLogging + metadata: + name: instance + namespace: openshift-logging + spec: + collection: + logs: + type: fluentd + fluentd: {} + ``` + + For vector: + + ```yaml + apiVersion: logging.openshift.io/v1 + kind: ClusterLogging + metadata: + name: instance + namespace: openshift-logging + spec: + collection: + logs: + type: vector + fluentd: {} + ``` + +* By default, TLS is enabled on all components deployed by loki-operator. Because the service certificates are signed by a cluster-internal CA you need to set up a secret that enables the collector to validate the certificate returned by the distributor. The secret must exist in the openshift-logging namespace, and must have a key `ca-bundle.crt`. + + The CA certificate is part of a ConfigMap that gets created by loki-operator as part of the LokiStack. Unfortunately this ConfigMap can not be used directly and has to be converted to a Secret readable by the collector. + + Fetch the `ca-bundle.crt` using: + + ```console + kubectl -n openshift-logging get cm lokistack-dev-gateway-ca-bundle -o jsonpath="{.data.service-ca\.crt}" > + ``` + + where `` can be `ca_bundle.crt` and used directly to create secret in the next step. + +* Once secret is fetched, create a new secret file: + + ```console + kubectl -n openshift-logging create secret generic loki-distributor-ca \ + --from-file=ca-bundle.crt= + ``` + + where `` is the file path where the `ca_bundle.crt` was copied to. + +* Now create a ClusterLogForwarder CR to forward logs to LokiStack: + + ```yaml + apiVersion: logging.openshift.io/v1 + kind: ClusterLogForwarder + metadata: + name: instance + namespace: openshift-logging + spec: + outputs: + - name: loki-operator + type: loki + url: https://lokistack-dev-distributor-http.openshift-logging.svc:3100 + secret: + name: loki-distributor-ca + loki: + tenantKey: log_type + pipelines: + - name: send-logs + inputRefs: + - application + - audit + - infrastructure + outputRefs: + - loki-operator + ``` + + _Note:_ The `tenantKey: log_type` gets resolved as `application`, `audit` or `infrastructure` by fluentd and vector based on the type of logs being collected. This is later used as different tenants for storing logs in Loki. + +## Troubleshooting + +### Log Entries Out of Order + +If the forwarder is configured to send too much data in a short span of time, Loki will back-pressure the forwarder and respond to the POST requests with `429` errors. +In order to alleviate this, follow this [documentation](./forwarding_logs_to_gateway.md#log-entries-out-of-order). diff --git a/operator/index.md b/operator/index.md index a9a5c11f2a7b..57dd92f19a3f 100644 --- a/operator/index.md +++ b/operator/index.md @@ -9,12 +9,19 @@ This is the Kubernetes Operator for Loki provided by the Grafana Loki SIG operat * There is also a [basic troubleshooting guide](https://github.com/grafana/loki/blob/master/operator/docs/hack_loki_operator.md#basic-troubleshooting-on-hacking-on-loki-operator) if you run into some common problems. * There is also a [document](https://github.com/grafana/loki/blob/master/operator/docs/hack_operator_make_run.md) which demonstrates how to use Loki Operator for development and testing locally without deploying the operator each time on Kind and OpenShift using the `make run` command. -### Sending Logs to Loki through the Gateway Component +### Sending Logs to Loki + +#### Sending Logs Through the Gateway Component * The [forwarding logs to LokiStack guide](https://github.com/grafana/loki/tree/master/operator/docs/forwarding_logs_to_gateway.md) provides instructions for configuring forwarding clients to ship logs to Loki through the gateway component. * This section details [how to connect a Promtail](https://github.com/grafana/loki/tree/master/operator/docs/forwarding_logs_to_gateway.md#promtail) installation to the gateway. * This section details [how to connect a Grafana Fluentd plugin](https://github.com/grafana/loki/tree/master/operator/docs/forwarding_logs_to_gateway.md#fluentd) installation to the gateway. +#### Sending Logs Directly to the Distributor Component + +* The [forwarding logs to LokiStack without LokiStack Gateway](https://github.com/grafana/loki/tree/master/operator/docs/forwarding_logs_without_gateway.md) is used to send application, infrastructure, and audit logs to the Loki Distributor as different tenants using Fluentd or Vector. +* The guide has a step-by-step guide to connect with OpenShift Logging for forwarding logs to LokiStack. + ### Installation of Storage Size Calculator on OpenShift * Storage size calculator works out of the box on OpenShift. For non-openshift distributions you will need to create services like prometheus, serviceMonitor, scrape configuration for log-file-metric exporter, promsecret to access the custom prometheus URL, token.