operator: Document how to connect to distributor component (#5551)

(cherry picked from commit 5e5067e)
grafana · Apr 7, 2022 · 6b1dc70 · 6b1dc70
1 parent 151d3a8
commit 6b1dc70
Show file tree

Hide file tree

Showing 3 changed files with 131 additions and 1 deletion.
diff --git a/operator/CHANGELOG.md b/operator/CHANGELOG.md
@@ -1,4 +1,13 @@
 ## Main
 
+- [5551](https://github.com/grafana/loki/pull/5551) **sasagarw**: Document how to connect to distributor component
+- [5624](https://github.com/grafana/loki/pull/5624) **periklis**: Use tenant name as id for mode openshift-logging (OpenShift)
+- [5621](https://github.com/grafana/loki/pull/5621) **periklis**: Use recommended labels for LokiStack components
+- [5607](https://github.com/grafana/loki/pull/5607) **periklis**: Use lokistack name as prefix for owned resources
+- [5588](https://github.com/grafana/loki/pull/5588) **periklis**: Add RBAC for Prometheus service discovery to Loki component metrics (OpenShift)
+- [5576](https://github.com/grafana/loki/pull/5576) **xperimental**: Change endpoints for generated liveness and readiness probes
+- [5560](https://github.com/grafana/loki/pull/5560) **periklis**: Fix service monitor's server name for operator metrics
+- [5345](https://github.com/grafana/loki/pull/5345) **ronensc**: Add flag to create Prometheus rules
+- [4974](https://github.com/grafana/loki/pull/5432) **Red-GV**: Provide storage configuration for Azure, GCS, and Swift through common_config
 - [4975](https://github.com/grafana/loki/pull/4975) **periklis**: Provide saner default for loki-operator managed chunk_target_size
 - [4974](https://github.com/grafana/loki/pull/5432) **Red-GV**: Provide storage configuration for Azure, GCS, and Swift through common_config
diff --git a/operator/docs/forwarding_logs_without_gateway.md b/operator/docs/forwarding_logs_without_gateway.md
@@ -0,0 +1,114 @@
+# Forwarding Logs to LokiStack without LokiStack Gateway
+
+This document describes how to send application, infrastructure, and audit logs to the Loki Distributor as different tenants using Fluentd or Vector.
+
+__Please read the [hacking guide](./hack_loki_operator.md) before proceeding with the following instructions.__
+
+_Note:_ This document only applies to OpenShift-based deployments.
+
+_Disclaimer:_ This document helps to connect the forwarder (fluentd or vector) to the LokiStack by going around the authentication gateway. This is not a normal configuration for an OpenShift-based deployments and should only be used for testing if going through the gateway is no option.
+
+## OpenShift Logging
+
+[OpenShift Logging](https://github.com/openshift/cluster-logging-operator) supports [forwarding logs to an external Loki instance](https://docs.openshift.com/container-platform/4.9/logging/cluster-logging-external.html#cluster-logging-collector-log-forward-loki_cluster-logging-external) using fluentd or vector as log forwarders.
+The below step-by-step guide will help you to send application, infrastructure, and audit logs to the LokiStack through the Distributor endpoint.
+The steps remain same for both fluentd and vector.
+
+In order to enable communication between the log forwarder and the distributor, follow these steps:
+
+* Deploy the Loki Operator and a `lokistack` instance for [OpenShift](./hack_loki_operator.md#hacking-on-loki-operator-on-openshift).
+
+* Deploy the OpenShift Logging Operator from the Operator Hub or using the following command locally:
+
+    ```console
+    make deploy-image deploy-catalog install
+    ```
+  
+* Create a Cluster Logging instance in the `openshift-logging` namespace with only `collection` defined.
+
+    For fluentd:
+
+    ```yaml
+    apiVersion: logging.openshift.io/v1
+    kind: ClusterLogging
+    metadata:
+      name: instance
+      namespace: openshift-logging
+    spec:
+      collection:
+        logs:
+          type: fluentd
+          fluentd: {}
+    ```
+
+    For vector:
+
+    ```yaml
+    apiVersion: logging.openshift.io/v1
+    kind: ClusterLogging
+    metadata:
+      name: instance
+      namespace: openshift-logging
+    spec:
+      collection:
+        logs:
+          type: vector
+          fluentd: {}
+    ```
+
+* By default, TLS is enabled on all components deployed by loki-operator. Because the service certificates are signed by a cluster-internal CA you need to set up a secret that enables the collector to validate the certificate returned by the distributor. The secret must exist in the openshift-logging namespace, and must have a key `ca-bundle.crt`.
+
+  The CA certificate is part of a ConfigMap that gets created by loki-operator as part of the LokiStack. Unfortunately this ConfigMap can not be used directly and has to be converted to a Secret readable by the collector.
+
+    Fetch the `ca-bundle.crt` using:
+
+    ```console
+    kubectl -n openshift-logging get cm lokistack-dev-gateway-ca-bundle -o jsonpath="{.data.service-ca\.crt}" > <FILE_NAME>
+    ```
+  
+    where `<FILE_NAME>` can be `ca_bundle.crt` and used directly to create secret in the next step.
+  
+* Once secret is fetched, create a new secret file:
+
+    ```console
+    kubectl -n openshift-logging create secret generic loki-distributor-ca \
+    --from-file=ca-bundle.crt=<PATH/TO/CA_BUNDLE.CRT>
+    ```
+    
+    where `<PATH/TO/CA_BUNDLE.CRT>` is the file path where the `ca_bundle.crt` was copied to.
+
+* Now create a ClusterLogForwarder CR to forward logs to LokiStack:
+
+  ```yaml
+  apiVersion: logging.openshift.io/v1
+  kind: ClusterLogForwarder
+  metadata:
+    name: instance
+    namespace: openshift-logging
+  spec:
+    outputs:
+     - name: loki-operator
+       type: loki
+       url: https://lokistack-dev-distributor-http.openshift-logging.svc:3100
+       secret:
+         name: loki-distributor-ca
+       loki:
+         tenantKey: log_type
+    pipelines:
+     - name: send-logs
+       inputRefs:
+       - application
+       - audit
+       - infrastructure
+       outputRefs:
+       - loki-operator
+  ```
+
+  _Note:_ The `tenantKey: log_type` gets resolved as `application`, `audit` or `infrastructure` by fluentd and vector based on the type of logs being collected. This is later used as different tenants for storing logs in Loki.
+
+## Troubleshooting
+
+### Log Entries Out of Order
+
+If the forwarder is configured to send too much data in a short span of time, Loki will back-pressure the forwarder and respond to the POST requests with `429` errors.
+In order to alleviate this, follow this [documentation](./forwarding_logs_to_gateway.md#log-entries-out-of-order).
diff --git a/operator/index.md b/operator/index.md
@@ -9,12 +9,19 @@ This is the Kubernetes Operator for Loki provided by the Grafana Loki SIG operat
 * There is also a [basic troubleshooting guide](https://github.com/grafana/loki/blob/master/operator/docs/hack_loki_operator.md#basic-troubleshooting-on-hacking-on-loki-operator) if you run into some common problems.
 * There is also a [document](https://github.com/grafana/loki/blob/master/operator/docs/hack_operator_make_run.md) which demonstrates how to use Loki Operator for development and testing locally without deploying the operator each time on Kind and OpenShift using the `make run` command.
 
-### Sending Logs to Loki through the Gateway Component
+### Sending Logs to Loki
+
+#### Sending Logs Through the Gateway Component
 
 * The [forwarding logs to LokiStack guide](https://github.com/grafana/loki/tree/master/operator/docs/forwarding_logs_to_gateway.md) provides instructions for configuring forwarding clients to ship logs to Loki through the gateway component.
 * This section details [how to connect a Promtail](https://github.com/grafana/loki/tree/master/operator/docs/forwarding_logs_to_gateway.md#promtail) installation to the gateway.
 * This section details [how to connect a Grafana Fluentd plugin](https://github.com/grafana/loki/tree/master/operator/docs/forwarding_logs_to_gateway.md#fluentd) installation to the gateway.
 
+#### Sending Logs Directly to the Distributor Component
+
+* The [forwarding logs to LokiStack without LokiStack Gateway](https://github.com/grafana/loki/tree/master/operator/docs/forwarding_logs_without_gateway.md) is used to send application, infrastructure, and audit logs to the Loki Distributor as different tenants using Fluentd or Vector.
+* The guide has a step-by-step guide to connect with OpenShift Logging for forwarding logs to LokiStack.
+
 ### Installation of Storage Size Calculator on OpenShift
 
 * Storage size calculator works out of the box on OpenShift. For non-openshift distributions you will need to create services like prometheus, serviceMonitor, scrape configuration for log-file-metric exporter, promsecret to access the custom prometheus URL, token.