From 5ec06bf439518c2a81b2d6602ff9cf671dcff2ef Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Thu, 30 May 2024 15:45:10 +0100 Subject: [PATCH 01/12] feat: Updated SS and microservices deployment docs --- .../helm/install-microservices/_index.md | 223 ++++++++++++++++++ .../testing-deployment-with-kind.md | 176 ++++++++++++++ .../install/helm/install-scalable/_index.md | 163 +++++++++++-- 3 files changed, 538 insertions(+), 24 deletions(-) create mode 100644 docs/sources/setup/install/helm/install-microservices/_index.md create mode 100644 docs/sources/setup/install/helm/install-microservices/testing-deployment-with-kind.md diff --git a/docs/sources/setup/install/helm/install-microservices/_index.md b/docs/sources/setup/install/helm/install-microservices/_index.md new file mode 100644 index 000000000000..9d661b35c23b --- /dev/null +++ b/docs/sources/setup/install/helm/install-microservices/_index.md @@ -0,0 +1,223 @@ +--- +title: Install the microservice Helm chart +menuTitle: Install microservice Loki +description: Installing Loki in microservice (distributed) mode using the Helm chart. +aliases: + - ../../../installation/helm/microservice/ + - ../../../installation/helm/install-microservice/ +weight: 300 +keywords: +--- + +# Install the Microservice Helm chart + + + +This Helm Chart deploys Grafana Loki on Kubernetes. + +This chart configures Loki to run Loki in [microservice / distributed mode]({{< relref "../../../../get-started/deployment-modes#microservices-mode" >}}), highly available architecture designed to work with AWS S3 object storage. + +The default Helm chart deploys the following components: +- **Ingester component** (3 replicas): Handles ingestion of data. +- **Querier component** (3 replicas, maxUnavailable: 2): Processes queries. Up to 2 replicas can be unavailable during updates. +- **QueryFrontend component** (2 replicas, maxUnavailable: 1): Manages frontend queries. Up to 1 replica can be unavailable during updates. +- **QueryScheduler component** (2 replicas): Schedules queries. +- **Distributor component** (3 replicas, maxUnavailable: 2): Distributes incoming requests. Up to 2 replicas can be unavailable during updates. +- **Compactor component** (1 replica): Compacts and processes stored data. +- **IndexGateway component** (2 replicas, maxUnavailable: 1): Handles indexing. Up to 1 replica can be unavailable during updates. + + + + +It is not recommended to run microservices mode with `filesystem` storage. + +**Prerequisites** + +- Helm 3 or above. See [Installing Helm](https://helm.sh/docs/intro/install/). +- A running Kubernetes cluster. +- (Optional) A Memcached deployment for better query performance. For information on configuring Memcached, refer to [caching section]({{< relref "../../../../operations/caching" >}}). + + +**To deploy Loki in simple scalable mode:** + + +1. Add [Grafana's chart repository](https://github.com/grafana/helm-charts) to Helm: + + ```bash + helm repo add grafana https://grafana.github.io/helm-charts + ``` + +1. Update the chart repository: + + ```bash + helm repo update + ``` + +1. Configure the object storage: + + - Create the configuration file `values.yaml`. The example below illustrates how to deploy Loki in test mode using MinIO as storage: + + ```yaml + loki: + schemaConfig: + configs: + - from: 2024-04-01 + store: tsdb + object_store: s3 + schema: v13 + index: + prefix: loki_index_ + period: 24h + ingester: + chunk_encoding: snappy + tracing: + enabled: true + querier: + # Default is 4, if you have enough memory and CPU you can increase, reduce if OOMing + max_concurrent: 4 + + #gateway: + # ingress: + # enabled: true + # hosts: + # - host: FIXME + # paths: + # - path: / + # pathType: Prefix + + deploymentMode: Distributed + + ingester: + replicas: 3 + querier: + replicas: 3 + maxUnavailable: 2 + queryFrontend: + replicas: 2 + maxUnavailable: 1 + queryScheduler: + replicas: 2 + distributor: + replicas: 3 + maxUnavailable: 2 + compactor: + replicas: 1 + indexGateway: + replicas: 2 + maxUnavailable: 1 + + bloomCompactor: + replicas: 0 + bloomGateway: + replicas: 0 + + # Enable minio for storage + minio: + enabled: true + + # Zero out replica counts of other deployment modes + backend: + replicas: 0 + read: + replicas: 0 + write: + replicas: 0 + + singleBinary: + replicas: 0 + ``` + + To configure other storage providers, refer to the [Helm Chart Reference]({{< relref "../reference" >}}). + + +1. Install or upgrade the Loki deployment. + - To install: + ```bash + helm install --values values.yaml loki grafana/loki + ``` + - To upgrade: + ```bash + helm upgrade --values values.yaml loki grafana/loki + ``` + + +1. Verify that Loki is running: + ```bash + kubectl get pods -n loki + ``` + +## AWS S3 Configuration + +To configure Loki to use AWS S3 as the object storage, you need to provide the following values in the `values.yaml` file: + +```yaml + loki: + schemaConfig: + configs: + - from: 2024-04-01 + store: tsdb + object_store: s3 + schema: v13 + index: + prefix: loki_index_ + period: 24h + ingester: + chunk_encoding: snappy + tracing: + enabled: true + querier: + max_concurrent: 4 + + storage: + type: s3 + bucketNames: + chunks: "chunks" + ruler: "ruler" + admin: "admin" + s3: + s3: + accessKeyId: + secretAccessKey: + s3ForcePathStyle: true + insecure: true + + deploymentMode: Distributed + + ingester: + replicas: 3 + querier: + replicas: 3 + maxUnavailable: 2 + queryFrontend: + replicas: 2 + maxUnavailable: 1 + queryScheduler: + replicas: 2 + distributor: + replicas: 3 + maxUnavailable: 2 + compactor: + replicas: 1 + indexGateway: + replicas: 2 + maxUnavailable: 1 + + bloomCompactor: + replicas: 0 + bloomGateway: + replicas: 0 + + backend: + replicas: 0 + read: + replicas: 0 + write: + replicas: 0 + + singleBinary: + replicas: 0 + +``` + +## Next Steps +Configure an agent to [send log data to Loki](/docs/loki//send-data/). diff --git a/docs/sources/setup/install/helm/install-microservices/testing-deployment-with-kind.md b/docs/sources/setup/install/helm/install-microservices/testing-deployment-with-kind.md new file mode 100644 index 000000000000..8cba9fa5bdc5 --- /dev/null +++ b/docs/sources/setup/install/helm/install-microservices/testing-deployment-with-kind.md @@ -0,0 +1,176 @@ +--- +title: Test the microservice Helm chart with Kind +menuTitle: Test the microservice Helm chart with Kind +description: Installing Loki in microservice (distributed) mode using the Helm chart. +aliases: + - ../../../installation/helm/test-microservice/ + - ../../../installation/helm/test-install-microservice/ +weight: 300 +keywords: +--- + + +# Test the Microservice Helm chart with Kind + + + +This guide walks you through testing the Grafana Loki Helm chart in microservice mode using [Kind](https://kind.sigs.k8s.io/), a tool for running local Kubernetes clusters using Docker container nodes. + +## Prerequisites + +To run through this tutorial, you need the following tools installed: +- [Docker](https://docs.docker.com/get-docker/) +- [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/) +- [Helm](https://helm.sh/docs/intro/install/) + +## Steps + +1. Create a Kind cluster config. Save the following configuration to a file named `kind-config.yaml`: + + ```yaml + kind: Cluster + apiVersion: kind.x-k8s.io/v1alpha4 + nodes: + - role: control-plane + - role: worker + - role: worker + ``` + +2. Create a Kind cluster using the configuration file: + + ```bash + kind create cluster --config kind-config.yaml + ``` + +3. Add the Grafana Helm chart repository to Helm: + + ```bash + helm repo add grafana https://grafana.github.io/helm-charts + ``` + +4. Update the chart repository: + + ```bash + helm repo update + ``` + +5. Create a `values.yaml` file to configure the Loki installation. The example below illustrates how to deploy Loki in test mode using MinIO as storage: + + ```yaml + loki: + schemaConfig: + configs: + - from: 2024-04-01 + store: tsdb + object_store: s3 + schema: v13 + index: + prefix: loki_index_ + period: 24h + ingester: + chunk_encoding: snappy + tracing: + enabled: true + querier: + # Default is 4, if you have enough memory and CPU you can increase, reduce if OOMing + max_concurrent: 4 + + #gateway: + # ingress: + # enabled: true + # hosts: + # - host: FIXME + # paths: + # - path: / + # pathType: Prefix + + deploymentMode: Distributed + + ingester: + replicas: 3 + querier: + replicas: 3 + maxUnavailable: 2 + queryFrontend: + replicas: 2 + maxUnavailable: 1 + queryScheduler: + replicas: 2 + distributor: + replicas: 3 + maxUnavailable: 2 + compactor: + replicas: 1 + indexGateway: + replicas: 2 + maxUnavailable: 1 + + bloomCompactor: + replicas: 0 + bloomGateway: + replicas: 0 + + # Enable minio for storage + minio: + enabled: true + + # Zero out replica counts of other deployment modes + backend: + replicas: 0 + read: + replicas: 0 + write: + replicas: 0 + + singleBinary: + replicas: 0 + ``` + +6. Install or upgrade the Loki deployment. + - To install: + ```bash + helm install --values values.yaml loki grafana/loki + ``` + - To upgrade: + ```bash + helm upgrade --values values.yaml loki grafana/loki + ``` + +7. Verify that Loki is running: + + ```bash + kubectl get pods + ``` + The output should an output similar to the following: + + ```bash + + loki-canary-8thrx 1/1 Running 0 167m + loki-canary-h965l 1/1 Running 0 167m + loki-canary-th8kb 1/1 Running 0 167m + loki-chunks-cache-0 0/2 Pending 0 167m + loki-compactor-0 1/1 Running 0 167m + loki-compactor-1 1/1 Running 0 167m + loki-distributor-7c9bb8f4dd-bcwc5 1/1 Running 0 167m + loki-distributor-7c9bb8f4dd-jh9h8 1/1 Running 0 167m + loki-distributor-7c9bb8f4dd-np5dw 1/1 Running 0 167m + loki-gateway-77bc447887-qgc56 1/1 Running 0 167m + loki-index-gateway-0 1/1 Running 0 167m + loki-index-gateway-1 1/1 Running 0 166m + loki-ingester-zone-a-0 1/1 Running 0 167m + loki-ingester-zone-b-0 1/1 Running 0 167m + loki-ingester-zone-c-0 1/1 Running 0 167m + loki-minio-0 1/1 Running 0 167m + loki-querier-bb8695c6d-bv9x2 1/1 Running 0 167m + loki-querier-bb8695c6d-bz2rw 1/1 Running 0 167m + loki-querier-bb8695c6d-z9qf8 1/1 Running 0 167m + loki-query-frontend-6659566b49-528j5 1/1 Running 0 167m + loki-query-frontend-6659566b49-84jtx 1/1 Running 0 167m + loki-query-frontend-6659566b49-9wfr7 1/1 Running 0 167m + loki-query-scheduler-f6dc4b949-fknfk 1/1 Running 0 167m + loki-query-scheduler-f6dc4b949-h4nwh 1/1 Running 0 167m + loki-query-scheduler-f6dc4b949-scfwp 1/1 Running 0 167m + loki-results-cache-0 2/2 Running 0 167m + + ``` + diff --git a/docs/sources/setup/install/helm/install-scalable/_index.md b/docs/sources/setup/install/helm/install-scalable/_index.md index 69e54ec1c319..f97ebe4c5666 100644 --- a/docs/sources/setup/install/helm/install-scalable/_index.md +++ b/docs/sources/setup/install/helm/install-scalable/_index.md @@ -24,7 +24,7 @@ The default Helm chart deploys the following components: - Loki Canary (1 DaemonSet) - Gateway (1 NGINX replica) - Minio (optional, if `minio.enabled=true`) -- Grafana Agent Operator + Grafana Agent (1 DaemonSet) - configured to monitor the Loki application. + @@ -37,7 +37,7 @@ It is not recommended to run scalable mode with `filesystem` storage. - (Optional) A Memcached deployment for better query performance. For information on configuring Memcached, refer to [caching section]({{< relref "../../../../operations/caching" >}}). -**To deploy Loki in simple scalable mode:** +**To deploy Loki in microservices mode:** 1. Add [Grafana's chart repository](https://github.com/grafana/helm-charts) to Helm: @@ -46,40 +46,83 @@ It is not recommended to run scalable mode with `filesystem` storage. helm repo add grafana https://grafana.github.io/helm-charts ``` -1. Update the chart repository: +2. Update the chart repository: ```bash helm repo update ``` -1. Configure the object storage: +3. Configure the object storage: - - Create the configuration file `values.yaml`. The example below illustrates a s3 configuration: + - Create the configuration file `values.yaml`. The example below illustrates how to deploy Loki in test mode using MinIO as storage: ```yaml - loki: - storage: - bucketNames: - chunks: chunks - ruler: ruler - admin: admin - type: s3 - s3: - endpoint: - region: - secretAccessKey: - accessKeyId: - s3ForcePathStyle: false - insecure: false + loki: + schemaConfig: + configs: + - from: 2024-04-01 + store: tsdb + object_store: s3 + schema: v13 + index: + prefix: loki_index_ + period: 24h + ingester: + chunk_encoding: snappy + tracing: + enabled: true + querier: + # Default is 4, if you have enough memory and CPU you can increase, reduce if OOMing + max_concurrent: 4 + + #gateway: + # ingress: + # enabled: true + # hosts: + # - host: FIXME + # paths: + # - path: / + # pathType: Prefix + + deploymentMode: SimpleScalable + + backend: + replicas: 3 + read: + replicas: 3 + write: + replicas: 3 + + # Enable minio for storage + minio: + enabled: true + + # Zero out replica counts of other deployment modes + singleBinary: + replicas: 0 + + ingester: + replicas: 0 + querier: + replicas: 0 + queryFrontend: + replicas: 0 + queryScheduler: + replicas: 0 + distributor: + replicas: 0 + compactor: + replicas: 0 + indexGateway: + replicas: 0 + bloomCompactor: + replicas: 0 + bloomGateway: + replicas: 0 ``` To configure other storage providers, refer to the [Helm Chart Reference]({{< relref "../reference" >}}). - - If you're just trying things, you can use the following configuration, that sets MinIO as storage: - ```yaml - minio: - enabled: true - ``` 1. Install or upgrade the Loki deployment. - To install: @@ -91,5 +134,77 @@ It is not recommended to run scalable mode with `filesystem` storage. helm upgrade --values values.yaml loki grafana/loki ``` +## AWS S3 Configuration + +To configure Loki to use AWS S3 as the object storage, you need to provide the following values in the `values.yaml` file: + +```yaml + loki: + schemaConfig: + configs: + - from: 2024-04-01 + store: tsdb + object_store: s3 + schema: v13 + index: + prefix: loki_index_ + period: 24h + ingester: + chunk_encoding: snappy + tracing: + enabled: true + querier: + max_concurrent: 4 + + storage: + type: s3 + bucketNames: + chunks: "chunks" + ruler: "ruler" + admin: "admin" + s3: + s3: + accessKeyId: + secretAccessKey: + s3ForcePathStyle: true + insecure: true + + deploymentMode: SimpleScalable + + backend: + replicas: 3 + read: + replicas: 3 + write: + replicas: 3 + + # Enable minio for storage + minio: + enabled: false + + # Zero out replica counts of other deployment modes + singleBinary: + replicas: 0 + + ingester: + replicas: 0 + querier: + replicas: 0 + queryFrontend: + replicas: 0 + queryScheduler: + replicas: 0 + distributor: + replicas: 0 + compactor: + replicas: 0 + indexGateway: + replicas: 0 + bloomCompactor: + replicas: 0 + bloomGateway: + replicas: 0 +``` + ## Next Steps Configure an agent to [send log data to Loki](/docs/loki//send-data/). From c8c209cae34fdaacbd0fbe5e5f6e6620c16ff458 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Thu, 30 May 2024 15:55:56 +0100 Subject: [PATCH 02/12] fix: fixed title --- docs/sources/setup/install/helm/install-microservices/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/setup/install/helm/install-microservices/_index.md b/docs/sources/setup/install/helm/install-microservices/_index.md index 9d661b35c23b..3688c839fba6 100644 --- a/docs/sources/setup/install/helm/install-microservices/_index.md +++ b/docs/sources/setup/install/helm/install-microservices/_index.md @@ -9,7 +9,7 @@ weight: 300 keywords: --- -# Install the Microservice Helm chart +# Install the microservice Helm chart From 45cb7ce609fdefeb28a69ee97756dec0d96bda97 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Thu, 30 May 2024 15:56:33 +0100 Subject: [PATCH 03/12] fix: fixed title --- .../helm/install-microservices/testing-deployment-with-kind.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/setup/install/helm/install-microservices/testing-deployment-with-kind.md b/docs/sources/setup/install/helm/install-microservices/testing-deployment-with-kind.md index 8cba9fa5bdc5..2052e0055913 100644 --- a/docs/sources/setup/install/helm/install-microservices/testing-deployment-with-kind.md +++ b/docs/sources/setup/install/helm/install-microservices/testing-deployment-with-kind.md @@ -10,7 +10,7 @@ keywords: --- -# Test the Microservice Helm chart with Kind +# Test the microservice Helm chart with Kind From 248171fc63579c0b388e7f74de5a6b1c4efb9746 Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Thu, 30 May 2024 16:38:14 +0100 Subject: [PATCH 04/12] fix: added endpoint --- docs/sources/setup/install/helm/install-microservices/_index.md | 1 + docs/sources/setup/install/helm/install-scalable/_index.md | 1 + 2 files changed, 2 insertions(+) diff --git a/docs/sources/setup/install/helm/install-microservices/_index.md b/docs/sources/setup/install/helm/install-microservices/_index.md index 3688c839fba6..19e4ee43a078 100644 --- a/docs/sources/setup/install/helm/install-microservices/_index.md +++ b/docs/sources/setup/install/helm/install-microservices/_index.md @@ -176,6 +176,7 @@ To configure Loki to use AWS S3 as the object storage, you need to provide the f admin: "admin" s3: s3: + endpoint: accessKeyId: secretAccessKey: s3ForcePathStyle: true diff --git a/docs/sources/setup/install/helm/install-scalable/_index.md b/docs/sources/setup/install/helm/install-scalable/_index.md index f97ebe4c5666..3e3a2ac8e6cf 100644 --- a/docs/sources/setup/install/helm/install-scalable/_index.md +++ b/docs/sources/setup/install/helm/install-scalable/_index.md @@ -164,6 +164,7 @@ To configure Loki to use AWS S3 as the object storage, you need to provide the f admin: "admin" s3: s3: + endpoint: accessKeyId: secretAccessKey: s3ForcePathStyle: true From 229d9539c018f2641914ee75e1344965d6c9eb89 Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Fri, 31 May 2024 11:20:56 +0100 Subject: [PATCH 05/12] feat: fixed based on review --- .../helm/install-microservices/_index.md | 75 ++++++++++++------- ...th-kind.md => deploy-locally-with-kind.md} | 14 ++-- .../install/helm/install-scalable/_index.md | 26 +++---- 3 files changed, 66 insertions(+), 49 deletions(-) rename docs/sources/setup/install/helm/install-microservices/{testing-deployment-with-kind.md => deploy-locally-with-kind.md} (90%) diff --git a/docs/sources/setup/install/helm/install-microservices/_index.md b/docs/sources/setup/install/helm/install-microservices/_index.md index 19e4ee43a078..5009da06a3c3 100644 --- a/docs/sources/setup/install/helm/install-microservices/_index.md +++ b/docs/sources/setup/install/helm/install-microservices/_index.md @@ -3,42 +3,35 @@ title: Install the microservice Helm chart menuTitle: Install microservice Loki description: Installing Loki in microservice (distributed) mode using the Helm chart. aliases: - - ../../../installation/helm/microservice/ - - ../../../installation/helm/install-microservice/ weight: 300 keywords: --- -# Install the microservice Helm chart - - +# Install the microservice Helm charts This Helm Chart deploys Grafana Loki on Kubernetes. -This chart configures Loki to run Loki in [microservice / distributed mode]({{< relref "../../../../get-started/deployment-modes#microservices-mode" >}}), highly available architecture designed to work with AWS S3 object storage. +This chart configures Loki to run Loki in [microservice / distributed mode]({{< relref "../../../../get-started/deployment-modes#microservices-mode" >}}). The microservices deployment mode runs components of Loki as distinct processes. The default Helm chart deploys the following components: +- **Compactor component** (1 replica): Compacts and processes stored data. +- **Distributor component** (3 replicas, maxUnavailable: 2): Distributes incoming requests. Up to 2 replicas can be unavailable during updates. +- **IndexGateway component** (2 replicas, maxUnavailable: 1): Handles indexing. Up to 1 replica can be unavailable during updates. - **Ingester component** (3 replicas): Handles ingestion of data. - **Querier component** (3 replicas, maxUnavailable: 2): Processes queries. Up to 2 replicas can be unavailable during updates. - **QueryFrontend component** (2 replicas, maxUnavailable: 1): Manages frontend queries. Up to 1 replica can be unavailable during updates. - **QueryScheduler component** (2 replicas): Schedules queries. -- **Distributor component** (3 replicas, maxUnavailable: 2): Distributes incoming requests. Up to 2 replicas can be unavailable during updates. -- **Compactor component** (1 replica): Compacts and processes stored data. -- **IndexGateway component** (2 replicas, maxUnavailable: 1): Handles indexing. Up to 1 replica can be unavailable during updates. - - - -It is not recommended to run microservices mode with `filesystem` storage. +It is not recommended to run scalable mode with `filesystem` storage. For the purpose of this guide, we will use MinIO as the object storage to provide a complete example. **Prerequisites** - Helm 3 or above. See [Installing Helm](https://helm.sh/docs/intro/install/). - A running Kubernetes cluster. -- (Optional) A Memcached deployment for better query performance. For information on configuring Memcached, refer to [caching section]({{< relref "../../../../operations/caching" >}}). +- (Optional) A Memcached deployment for better query performance. For information on configuring Memcached, refer to the [caching section](https://grafana.com/docs/loki//operations/caching/). -**To deploy Loki in simple scalable mode:** +**To deploy Loki in microservice mode (with MinIO):** 1. Add [Grafana's chart repository](https://github.com/grafana/helm-charts) to Helm: @@ -47,15 +40,13 @@ It is not recommended to run microservices mode with `filesystem` storage. helm repo add grafana https://grafana.github.io/helm-charts ``` -1. Update the chart repository: +2. Update the chart repository: ```bash helm repo update ``` -1. Configure the object storage: - - - Create the configuration file `values.yaml`. The example below illustrates how to deploy Loki in test mode using MinIO as storage: +3. Create the configuration file `values.yaml`. The example below illustrates how to deploy Loki in test mode using MinIO as storage: ```yaml loki: @@ -127,10 +118,7 @@ It is not recommended to run microservices mode with `filesystem` storage. replicas: 0 ``` - To configure other storage providers, refer to the [Helm Chart Reference]({{< relref "../reference" >}}). - - -1. Install or upgrade the Loki deployment. +4. Install or upgrade the Loki deployment. - To install: ```bash helm install --values values.yaml loki grafana/loki @@ -145,10 +133,40 @@ It is not recommended to run microservices mode with `filesystem` storage. ```bash kubectl get pods -n loki ``` + The output should an output similar to the following: + + ```bash + loki-canary-8thrx 1/1 Running 0 167m + loki-canary-h965l 1/1 Running 0 167m + loki-canary-th8kb 1/1 Running 0 167m + loki-chunks-cache-0 0/2 Pending 0 167m + loki-compactor-0 1/1 Running 0 167m + loki-compactor-1 1/1 Running 0 167m + loki-distributor-7c9bb8f4dd-bcwc5 1/1 Running 0 167m + loki-distributor-7c9bb8f4dd-jh9h8 1/1 Running 0 167m + loki-distributor-7c9bb8f4dd-np5dw 1/1 Running 0 167m + loki-gateway-77bc447887-qgc56 1/1 Running 0 167m + loki-index-gateway-0 1/1 Running 0 167m + loki-index-gateway-1 1/1 Running 0 166m + loki-ingester-zone-a-0 1/1 Running 0 167m + loki-ingester-zone-b-0 1/1 Running 0 167m + loki-ingester-zone-c-0 1/1 Running 0 167m + loki-minio-0 1/1 Running 0 167m + loki-querier-bb8695c6d-bv9x2 1/1 Running 0 167m + loki-querier-bb8695c6d-bz2rw 1/1 Running 0 167m + loki-querier-bb8695c6d-z9qf8 1/1 Running 0 167m + loki-query-frontend-6659566b49-528j5 1/1 Running 0 167m + loki-query-frontend-6659566b49-84jtx 1/1 Running 0 167m + loki-query-frontend-6659566b49-9wfr7 1/1 Running 0 167m + loki-query-scheduler-f6dc4b949-fknfk 1/1 Running 0 167m + loki-query-scheduler-f6dc4b949-h4nwh 1/1 Running 0 167m + loki-query-scheduler-f6dc4b949-scfwp 1/1 Running 0 167m + loki-results-cache-0 2/2 Running 0 167m + ``` ## AWS S3 Configuration -To configure Loki to use AWS S3 as the object storage, you need to provide the following values in the `values.yaml` file: +To configure Loki to use AWS S3 as the object storage rather than MinIO, you need to provide the following values in the `values.yaml` file: ```yaml loki: @@ -184,6 +202,10 @@ To configure Loki to use AWS S3 as the object storage, you need to provide the f deploymentMode: Distributed + # Disable minio storage + minio: + enabled: false + ingester: replicas: 3 querier: @@ -220,5 +242,8 @@ To configure Loki to use AWS S3 as the object storage, you need to provide the f ``` +To configure other storage providers, refer to the [Helm Chart Reference]({{< relref "../reference" >}}). + ## Next Steps -Configure an agent to [send log data to Loki](/docs/loki//send-data/). +* Configure an agent to [send log data to Loki](/docs/loki//send-data/). +* Monitor the Loki deployment using the [Meta Monitoring Healm chart](/docs/loki//monitor-and-alert/) diff --git a/docs/sources/setup/install/helm/install-microservices/testing-deployment-with-kind.md b/docs/sources/setup/install/helm/install-microservices/deploy-locally-with-kind.md similarity index 90% rename from docs/sources/setup/install/helm/install-microservices/testing-deployment-with-kind.md rename to docs/sources/setup/install/helm/install-microservices/deploy-locally-with-kind.md index 2052e0055913..32bb483514cb 100644 --- a/docs/sources/setup/install/helm/install-microservices/testing-deployment-with-kind.md +++ b/docs/sources/setup/install/helm/install-microservices/deploy-locally-with-kind.md @@ -1,20 +1,16 @@ --- -title: Test the microservice Helm chart with Kind -menuTitle: Test the microservice Helm chart with Kind -description: Installing Loki in microservice (distributed) mode using the Helm chart. +title: Deploy the Helm chart localy with Kind +menuTitle: Deploy the Helm chart localy with Kind +description: Installing Loki in microservice (distributed) with Kind for local development and testing purposes. aliases: - - ../../../installation/helm/test-microservice/ - - ../../../installation/helm/test-install-microservice/ weight: 300 keywords: --- -# Test the microservice Helm chart with Kind - - +# Deploy the Helm chart localy with Kind -This guide walks you through testing the Grafana Loki Helm chart in microservice mode using [Kind](https://kind.sigs.k8s.io/), a tool for running local Kubernetes clusters using Docker container nodes. +This guide walks you through testing the Grafana Loki Helm chart localy in microservice mode using [Kind](https://kind.sigs.k8s.io/), a tool for running local Kubernetes clusters using Docker container nodes. ## Prerequisites diff --git a/docs/sources/setup/install/helm/install-scalable/_index.md b/docs/sources/setup/install/helm/install-scalable/_index.md index 3e3a2ac8e6cf..c77d524f6db1 100644 --- a/docs/sources/setup/install/helm/install-scalable/_index.md +++ b/docs/sources/setup/install/helm/install-scalable/_index.md @@ -10,12 +10,10 @@ keywords: --- # Install the simple scalable Helm chart - - This Helm Chart deploys Grafana Loki on Kubernetes. -This chart configures Loki to run `read`, `write`, and `backend` targets in a [scalable mode]({{< relref "../../../../get-started/deployment-modes#simple-scalable" >}}), highly available architecture designed to work with AWS S3 object storage. The chart also supports self-monitoring or meta-monitoring by deploying Grafana Agent to monitor Loki itself, by scraping its metrics and logs. +This chart configures Loki to run `read`, `write`, and `backend` targets in a [scalable mode]({{< relref "../../../../get-started/deployment-modes#simple-scalable" >}}). Loki’s simple scalable deployment mode separates execution paths into read, write, and backend targets. The default Helm chart deploys the following components: - Read component (3 replicas) @@ -28,7 +26,7 @@ The default Helm chart deploys the following components: -It is not recommended to run scalable mode with `filesystem` storage. +It is not recommended to run scalable mode with `filesystem` storage. For the purpose of this guide, we will use MinIO as the object storage to provide a complete example. **Prerequisites** @@ -37,7 +35,7 @@ It is not recommended to run scalable mode with `filesystem` storage. - (Optional) A Memcached deployment for better query performance. For information on configuring Memcached, refer to [caching section]({{< relref "../../../../operations/caching" >}}). -**To deploy Loki in microservices mode:** +**To deploy Loki in simple scalable mode:** 1. Add [Grafana's chart repository](https://github.com/grafana/helm-charts) to Helm: @@ -52,9 +50,7 @@ It is not recommended to run scalable mode with `filesystem` storage. helm repo update ``` -3. Configure the object storage: - - - Create the configuration file `values.yaml`. The example below illustrates how to deploy Loki in test mode using MinIO as storage: +3. Create the configuration file `values.yaml`. The example below illustrates how to deploy Loki in test mode using MinIO as storage: ```yaml loki: @@ -121,10 +117,7 @@ It is not recommended to run scalable mode with `filesystem` storage. replicas: 0 ``` - To configure other storage providers, refer to the [Helm Chart Reference]({{< relref "../reference" >}}). - - -1. Install or upgrade the Loki deployment. +4. Install or upgrade the Loki deployment. - To install: ```bash helm install --values values.yaml loki grafana/loki @@ -136,7 +129,7 @@ It is not recommended to run scalable mode with `filesystem` storage. ## AWS S3 Configuration -To configure Loki to use AWS S3 as the object storage, you need to provide the following values in the `values.yaml` file: +To configure Loki to use AWS S3 as the object storage rather than MinIO, you need to provide the following values in the `values.yaml` file: ```yaml loki: @@ -179,7 +172,7 @@ To configure Loki to use AWS S3 as the object storage, you need to provide the f write: replicas: 3 - # Enable minio for storage + # Disable minio storage minio: enabled: false @@ -207,5 +200,8 @@ To configure Loki to use AWS S3 as the object storage, you need to provide the f replicas: 0 ``` +To configure other storage providers, refer to the [Helm Chart Reference]({{< relref "../reference" >}}). + ## Next Steps -Configure an agent to [send log data to Loki](/docs/loki//send-data/). +* Configure an agent to [send log data to Loki](/docs/loki//send-data/). +* Monitor the Loki deployment using the [Meta Monitoring Healm chart](/docs/loki//monitor-and-alert/) \ No newline at end of file From 31debcda829a34e1e7399a46d886c4444c01a829 Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Fri, 31 May 2024 11:21:30 +0100 Subject: [PATCH 06/12] feat: removed old comment --- docs/sources/setup/install/helm/install-scalable/_index.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/docs/sources/setup/install/helm/install-scalable/_index.md b/docs/sources/setup/install/helm/install-scalable/_index.md index c77d524f6db1..364240ebfe2b 100644 --- a/docs/sources/setup/install/helm/install-scalable/_index.md +++ b/docs/sources/setup/install/helm/install-scalable/_index.md @@ -23,9 +23,6 @@ The default Helm chart deploys the following components: - Gateway (1 NGINX replica) - Minio (optional, if `minio.enabled=true`) - - - It is not recommended to run scalable mode with `filesystem` storage. For the purpose of this guide, we will use MinIO as the object storage to provide a complete example. **Prerequisites** From ec182cf550d855f537df84f2550cc5aac7ffbc69 Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Fri, 31 May 2024 11:23:18 +0100 Subject: [PATCH 07/12] fix: fixed monitoring URL --- docs/sources/setup/install/helm/install-microservices/_index.md | 2 +- docs/sources/setup/install/helm/install-scalable/_index.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/sources/setup/install/helm/install-microservices/_index.md b/docs/sources/setup/install/helm/install-microservices/_index.md index 5009da06a3c3..2b3989f8ce63 100644 --- a/docs/sources/setup/install/helm/install-microservices/_index.md +++ b/docs/sources/setup/install/helm/install-microservices/_index.md @@ -246,4 +246,4 @@ To configure other storage providers, refer to the [Helm Chart Reference]({{< re ## Next Steps * Configure an agent to [send log data to Loki](/docs/loki//send-data/). -* Monitor the Loki deployment using the [Meta Monitoring Healm chart](/docs/loki//monitor-and-alert/) +* Monitor the Loki deployment using the [Meta Monitoring Healm chart](/docs/loki//setup/install/helm/monitor-and-alert/) diff --git a/docs/sources/setup/install/helm/install-scalable/_index.md b/docs/sources/setup/install/helm/install-scalable/_index.md index 364240ebfe2b..26d753ee2877 100644 --- a/docs/sources/setup/install/helm/install-scalable/_index.md +++ b/docs/sources/setup/install/helm/install-scalable/_index.md @@ -201,4 +201,4 @@ To configure other storage providers, refer to the [Helm Chart Reference]({{< re ## Next Steps * Configure an agent to [send log data to Loki](/docs/loki//send-data/). -* Monitor the Loki deployment using the [Meta Monitoring Healm chart](/docs/loki//monitor-and-alert/) \ No newline at end of file +* Monitor the Loki deployment using the [Meta Monitoring Healm chart](/docs/loki//setup/install/helm/monitor-and-alert/) \ No newline at end of file From a99fccaf121a5b45c71e2e1569ca859899bcad96 Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Fri, 31 May 2024 11:25:32 +0100 Subject: [PATCH 08/12] fix: revert deleted file --- .../testing-deployment-with-kind.md | 172 ++++++++++++++++++ 1 file changed, 172 insertions(+) create mode 100644 docs/sources/setup/install/helm/install-microservices/testing-deployment-with-kind.md diff --git a/docs/sources/setup/install/helm/install-microservices/testing-deployment-with-kind.md b/docs/sources/setup/install/helm/install-microservices/testing-deployment-with-kind.md new file mode 100644 index 000000000000..32bb483514cb --- /dev/null +++ b/docs/sources/setup/install/helm/install-microservices/testing-deployment-with-kind.md @@ -0,0 +1,172 @@ +--- +title: Deploy the Helm chart localy with Kind +menuTitle: Deploy the Helm chart localy with Kind +description: Installing Loki in microservice (distributed) with Kind for local development and testing purposes. +aliases: +weight: 300 +keywords: +--- + + +# Deploy the Helm chart localy with Kind + +This guide walks you through testing the Grafana Loki Helm chart localy in microservice mode using [Kind](https://kind.sigs.k8s.io/), a tool for running local Kubernetes clusters using Docker container nodes. + +## Prerequisites + +To run through this tutorial, you need the following tools installed: +- [Docker](https://docs.docker.com/get-docker/) +- [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/) +- [Helm](https://helm.sh/docs/intro/install/) + +## Steps + +1. Create a Kind cluster config. Save the following configuration to a file named `kind-config.yaml`: + + ```yaml + kind: Cluster + apiVersion: kind.x-k8s.io/v1alpha4 + nodes: + - role: control-plane + - role: worker + - role: worker + ``` + +2. Create a Kind cluster using the configuration file: + + ```bash + kind create cluster --config kind-config.yaml + ``` + +3. Add the Grafana Helm chart repository to Helm: + + ```bash + helm repo add grafana https://grafana.github.io/helm-charts + ``` + +4. Update the chart repository: + + ```bash + helm repo update + ``` + +5. Create a `values.yaml` file to configure the Loki installation. The example below illustrates how to deploy Loki in test mode using MinIO as storage: + + ```yaml + loki: + schemaConfig: + configs: + - from: 2024-04-01 + store: tsdb + object_store: s3 + schema: v13 + index: + prefix: loki_index_ + period: 24h + ingester: + chunk_encoding: snappy + tracing: + enabled: true + querier: + # Default is 4, if you have enough memory and CPU you can increase, reduce if OOMing + max_concurrent: 4 + + #gateway: + # ingress: + # enabled: true + # hosts: + # - host: FIXME + # paths: + # - path: / + # pathType: Prefix + + deploymentMode: Distributed + + ingester: + replicas: 3 + querier: + replicas: 3 + maxUnavailable: 2 + queryFrontend: + replicas: 2 + maxUnavailable: 1 + queryScheduler: + replicas: 2 + distributor: + replicas: 3 + maxUnavailable: 2 + compactor: + replicas: 1 + indexGateway: + replicas: 2 + maxUnavailable: 1 + + bloomCompactor: + replicas: 0 + bloomGateway: + replicas: 0 + + # Enable minio for storage + minio: + enabled: true + + # Zero out replica counts of other deployment modes + backend: + replicas: 0 + read: + replicas: 0 + write: + replicas: 0 + + singleBinary: + replicas: 0 + ``` + +6. Install or upgrade the Loki deployment. + - To install: + ```bash + helm install --values values.yaml loki grafana/loki + ``` + - To upgrade: + ```bash + helm upgrade --values values.yaml loki grafana/loki + ``` + +7. Verify that Loki is running: + + ```bash + kubectl get pods + ``` + The output should an output similar to the following: + + ```bash + + loki-canary-8thrx 1/1 Running 0 167m + loki-canary-h965l 1/1 Running 0 167m + loki-canary-th8kb 1/1 Running 0 167m + loki-chunks-cache-0 0/2 Pending 0 167m + loki-compactor-0 1/1 Running 0 167m + loki-compactor-1 1/1 Running 0 167m + loki-distributor-7c9bb8f4dd-bcwc5 1/1 Running 0 167m + loki-distributor-7c9bb8f4dd-jh9h8 1/1 Running 0 167m + loki-distributor-7c9bb8f4dd-np5dw 1/1 Running 0 167m + loki-gateway-77bc447887-qgc56 1/1 Running 0 167m + loki-index-gateway-0 1/1 Running 0 167m + loki-index-gateway-1 1/1 Running 0 166m + loki-ingester-zone-a-0 1/1 Running 0 167m + loki-ingester-zone-b-0 1/1 Running 0 167m + loki-ingester-zone-c-0 1/1 Running 0 167m + loki-minio-0 1/1 Running 0 167m + loki-querier-bb8695c6d-bv9x2 1/1 Running 0 167m + loki-querier-bb8695c6d-bz2rw 1/1 Running 0 167m + loki-querier-bb8695c6d-z9qf8 1/1 Running 0 167m + loki-query-frontend-6659566b49-528j5 1/1 Running 0 167m + loki-query-frontend-6659566b49-84jtx 1/1 Running 0 167m + loki-query-frontend-6659566b49-9wfr7 1/1 Running 0 167m + loki-query-scheduler-f6dc4b949-fknfk 1/1 Running 0 167m + loki-query-scheduler-f6dc4b949-h4nwh 1/1 Running 0 167m + loki-query-scheduler-f6dc4b949-scfwp 1/1 Running 0 167m + loki-results-cache-0 2/2 Running 0 167m + + ``` + From 7194db635a68b78c1528dae8a402628fe0732aec Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Fri, 31 May 2024 11:26:25 +0100 Subject: [PATCH 09/12] fix: naming --- docs/sources/setup/install/helm/install-microservices/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/setup/install/helm/install-microservices/_index.md b/docs/sources/setup/install/helm/install-microservices/_index.md index 2b3989f8ce63..7826ed598e4f 100644 --- a/docs/sources/setup/install/helm/install-microservices/_index.md +++ b/docs/sources/setup/install/helm/install-microservices/_index.md @@ -7,7 +7,7 @@ weight: 300 keywords: --- -# Install the microservice Helm charts +# Install the microservice Helm chart This Helm Chart deploys Grafana Loki on Kubernetes. From 7a63d3a5d4191ce225cb89828199a78ec452462e Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Fri, 31 May 2024 11:34:01 +0100 Subject: [PATCH 10/12] fix: fixed aliases and removed old file --- .../helm/install-microservices/_index.md | 1 - .../deploy-locally-with-kind.md | 1 - .../testing-deployment-with-kind.md | 172 ------------------ 3 files changed, 174 deletions(-) delete mode 100644 docs/sources/setup/install/helm/install-microservices/testing-deployment-with-kind.md diff --git a/docs/sources/setup/install/helm/install-microservices/_index.md b/docs/sources/setup/install/helm/install-microservices/_index.md index 7826ed598e4f..8dc67e1762de 100644 --- a/docs/sources/setup/install/helm/install-microservices/_index.md +++ b/docs/sources/setup/install/helm/install-microservices/_index.md @@ -2,7 +2,6 @@ title: Install the microservice Helm chart menuTitle: Install microservice Loki description: Installing Loki in microservice (distributed) mode using the Helm chart. -aliases: weight: 300 keywords: --- diff --git a/docs/sources/setup/install/helm/install-microservices/deploy-locally-with-kind.md b/docs/sources/setup/install/helm/install-microservices/deploy-locally-with-kind.md index 32bb483514cb..e064342b9cec 100644 --- a/docs/sources/setup/install/helm/install-microservices/deploy-locally-with-kind.md +++ b/docs/sources/setup/install/helm/install-microservices/deploy-locally-with-kind.md @@ -2,7 +2,6 @@ title: Deploy the Helm chart localy with Kind menuTitle: Deploy the Helm chart localy with Kind description: Installing Loki in microservice (distributed) with Kind for local development and testing purposes. -aliases: weight: 300 keywords: --- diff --git a/docs/sources/setup/install/helm/install-microservices/testing-deployment-with-kind.md b/docs/sources/setup/install/helm/install-microservices/testing-deployment-with-kind.md deleted file mode 100644 index 32bb483514cb..000000000000 --- a/docs/sources/setup/install/helm/install-microservices/testing-deployment-with-kind.md +++ /dev/null @@ -1,172 +0,0 @@ ---- -title: Deploy the Helm chart localy with Kind -menuTitle: Deploy the Helm chart localy with Kind -description: Installing Loki in microservice (distributed) with Kind for local development and testing purposes. -aliases: -weight: 300 -keywords: ---- - - -# Deploy the Helm chart localy with Kind - -This guide walks you through testing the Grafana Loki Helm chart localy in microservice mode using [Kind](https://kind.sigs.k8s.io/), a tool for running local Kubernetes clusters using Docker container nodes. - -## Prerequisites - -To run through this tutorial, you need the following tools installed: -- [Docker](https://docs.docker.com/get-docker/) -- [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/) -- [Helm](https://helm.sh/docs/intro/install/) - -## Steps - -1. Create a Kind cluster config. Save the following configuration to a file named `kind-config.yaml`: - - ```yaml - kind: Cluster - apiVersion: kind.x-k8s.io/v1alpha4 - nodes: - - role: control-plane - - role: worker - - role: worker - ``` - -2. Create a Kind cluster using the configuration file: - - ```bash - kind create cluster --config kind-config.yaml - ``` - -3. Add the Grafana Helm chart repository to Helm: - - ```bash - helm repo add grafana https://grafana.github.io/helm-charts - ``` - -4. Update the chart repository: - - ```bash - helm repo update - ``` - -5. Create a `values.yaml` file to configure the Loki installation. The example below illustrates how to deploy Loki in test mode using MinIO as storage: - - ```yaml - loki: - schemaConfig: - configs: - - from: 2024-04-01 - store: tsdb - object_store: s3 - schema: v13 - index: - prefix: loki_index_ - period: 24h - ingester: - chunk_encoding: snappy - tracing: - enabled: true - querier: - # Default is 4, if you have enough memory and CPU you can increase, reduce if OOMing - max_concurrent: 4 - - #gateway: - # ingress: - # enabled: true - # hosts: - # - host: FIXME - # paths: - # - path: / - # pathType: Prefix - - deploymentMode: Distributed - - ingester: - replicas: 3 - querier: - replicas: 3 - maxUnavailable: 2 - queryFrontend: - replicas: 2 - maxUnavailable: 1 - queryScheduler: - replicas: 2 - distributor: - replicas: 3 - maxUnavailable: 2 - compactor: - replicas: 1 - indexGateway: - replicas: 2 - maxUnavailable: 1 - - bloomCompactor: - replicas: 0 - bloomGateway: - replicas: 0 - - # Enable minio for storage - minio: - enabled: true - - # Zero out replica counts of other deployment modes - backend: - replicas: 0 - read: - replicas: 0 - write: - replicas: 0 - - singleBinary: - replicas: 0 - ``` - -6. Install or upgrade the Loki deployment. - - To install: - ```bash - helm install --values values.yaml loki grafana/loki - ``` - - To upgrade: - ```bash - helm upgrade --values values.yaml loki grafana/loki - ``` - -7. Verify that Loki is running: - - ```bash - kubectl get pods - ``` - The output should an output similar to the following: - - ```bash - - loki-canary-8thrx 1/1 Running 0 167m - loki-canary-h965l 1/1 Running 0 167m - loki-canary-th8kb 1/1 Running 0 167m - loki-chunks-cache-0 0/2 Pending 0 167m - loki-compactor-0 1/1 Running 0 167m - loki-compactor-1 1/1 Running 0 167m - loki-distributor-7c9bb8f4dd-bcwc5 1/1 Running 0 167m - loki-distributor-7c9bb8f4dd-jh9h8 1/1 Running 0 167m - loki-distributor-7c9bb8f4dd-np5dw 1/1 Running 0 167m - loki-gateway-77bc447887-qgc56 1/1 Running 0 167m - loki-index-gateway-0 1/1 Running 0 167m - loki-index-gateway-1 1/1 Running 0 166m - loki-ingester-zone-a-0 1/1 Running 0 167m - loki-ingester-zone-b-0 1/1 Running 0 167m - loki-ingester-zone-c-0 1/1 Running 0 167m - loki-minio-0 1/1 Running 0 167m - loki-querier-bb8695c6d-bv9x2 1/1 Running 0 167m - loki-querier-bb8695c6d-bz2rw 1/1 Running 0 167m - loki-querier-bb8695c6d-z9qf8 1/1 Running 0 167m - loki-query-frontend-6659566b49-528j5 1/1 Running 0 167m - loki-query-frontend-6659566b49-84jtx 1/1 Running 0 167m - loki-query-frontend-6659566b49-9wfr7 1/1 Running 0 167m - loki-query-scheduler-f6dc4b949-fknfk 1/1 Running 0 167m - loki-query-scheduler-f6dc4b949-h4nwh 1/1 Running 0 167m - loki-query-scheduler-f6dc4b949-scfwp 1/1 Running 0 167m - loki-results-cache-0 2/2 Running 0 167m - - ``` - From 981360a58dbedcb98cbac7176a98584231793224 Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Mon, 3 Jun 2024 11:09:54 +0100 Subject: [PATCH 11/12] Added azure and s3 examples --- .../helm/install-microservices/_index.md | 116 ++++++++++++++++-- .../install/helm/install-scalable/_index.md | 112 +++++++++++++++-- 2 files changed, 210 insertions(+), 18 deletions(-) diff --git a/docs/sources/setup/install/helm/install-microservices/_index.md b/docs/sources/setup/install/helm/install-microservices/_index.md index 8dc67e1762de..71f94673fe53 100644 --- a/docs/sources/setup/install/helm/install-microservices/_index.md +++ b/docs/sources/setup/install/helm/install-microservices/_index.md @@ -163,11 +163,15 @@ It is not recommended to run scalable mode with `filesystem` storage. For the pu loki-results-cache-0 2/2 Running 0 167m ``` -## AWS S3 Configuration +## Object Storage Configuration -To configure Loki to use AWS S3 as the object storage rather than MinIO, you need to provide the following values in the `values.yaml` file: +After testing Loki with MinIO, it is recommended to configure Loki with an object storage provider. The following examples shows how to configure Loki with different object storage providers: + +{{< code >}} + +```s3 +# Example configuration for Loki with S3 storage -```yaml loki: schemaConfig: configs: @@ -192,12 +196,24 @@ To configure Loki to use AWS S3 as the object storage rather than MinIO, you nee ruler: "ruler" admin: "admin" s3: - s3: - endpoint: - accessKeyId: - secretAccessKey: - s3ForcePathStyle: true - insecure: true + # s3 URL can be used to specify the endpoint, access key, secret key, and bucket name + s3: s3://access_key:secret_access_key@custom_endpoint/bucket_name + # AWS endpoint URL + endpoint: + # AWS region where the S3 bucket is located + region: + # AWS secret access key + secretAccessKey: + # AWS access key ID + accessKeyId: + # AWS signature version (e.g., v2 or v4) + signatureVersion: + # Forces the path style for S3 (true/false) + s3ForcePathStyle: false + # Allows insecure (HTTP) connections (true/false) + insecure: false + # HTTP configuration settings + http_config: {} deploymentMode: Distributed @@ -241,6 +257,88 @@ To configure Loki to use AWS S3 as the object storage rather than MinIO, you nee ``` +```azure +# Example configuration for Loki with Azure Blob Storage + +loki: + schemaConfig: + configs: + - from: 2024-04-01 + store: tsdb + object_store: azure + schema: v13 + index: + prefix: loki_index_ + period: 24h + ingester: + chunk_encoding: snappy + tracing: + enabled: true + querier: + max_concurrent: 4 + + storage: + type: azure + azure: + # Name of the Azure Blob Storage account + accountName: + # Key associated with the Azure Blob Storage account + accountKey: + # Comprehensive connection string for Azure Blob Storage account (Can be used to replace endpoint, accountName, and accountKey) + connectionString: + # Flag indicating whether to use Azure Managed Identity for authentication + useManagedIdentity: false + # Flag indicating whether to use a federated token for authentication + useFederatedToken: false + # Client ID of the user-assigned managed identity (if applicable) + userAssignedId: + # Timeout duration for requests made to the Azure Blob Storage account (in seconds) + requestTimeout: + # Domain suffix of the Azure Blob Storage service endpoint (e.g., core.windows.net) + endpointSuffix: + bucketNames: + chunks: "chunks" + ruler: "ruler" + admin: "admin" +deploymentMode: Distributed + +ingester: + replicas: 3 +querier: + replicas: 3 + maxUnavailable: 2 +queryFrontend: + replicas: 2 + maxUnavailable: 1 +queryScheduler: + replicas: 2 +distributor: + replicas: 3 + maxUnavailable: 2 +compactor: + replicas: 1 +indexGateway: + replicas: 2 + maxUnavailable: 1 + +bloomCompactor: + replicas: 0 +bloomGateway: + replicas: 0 + +backend: + replicas: 0 +read: + replicas: 0 +write: + replicas: 0 + +singleBinary: + replicas: 0 + +``` +{{< /code >}} + To configure other storage providers, refer to the [Helm Chart Reference]({{< relref "../reference" >}}). ## Next Steps diff --git a/docs/sources/setup/install/helm/install-scalable/_index.md b/docs/sources/setup/install/helm/install-scalable/_index.md index 26d753ee2877..e27f544b28f0 100644 --- a/docs/sources/setup/install/helm/install-scalable/_index.md +++ b/docs/sources/setup/install/helm/install-scalable/_index.md @@ -124,11 +124,13 @@ It is not recommended to run scalable mode with `filesystem` storage. For the pu helm upgrade --values values.yaml loki grafana/loki ``` -## AWS S3 Configuration +## Object Storage Configuration -To configure Loki to use AWS S3 as the object storage rather than MinIO, you need to provide the following values in the `values.yaml` file: +After testing Loki with MinIO, it is recommended to configure Loki with an object storage provider. The following examples shows how to configure Loki with different object storage providers: -```yaml +{{< code >}} + +```s3 loki: schemaConfig: configs: @@ -153,12 +155,102 @@ To configure Loki to use AWS S3 as the object storage rather than MinIO, you nee ruler: "ruler" admin: "admin" s3: - s3: - endpoint: - accessKeyId: - secretAccessKey: - s3ForcePathStyle: true - insecure: true + # s3 URL can be used to specify the endpoint, access key, secret key, and bucket name + s3: s3://access_key:secret_access_key@custom_endpoint/bucket_name + # AWS endpoint URL + endpoint: + # AWS region where the S3 bucket is located + region: + # AWS secret access key + secretAccessKey: + # AWS access key ID + accessKeyId: + # AWS signature version (e.g., v2 or v4) + signatureVersion: + # Forces the path style for S3 (true/false) + s3ForcePathStyle: false + # Allows insecure (HTTP) connections (true/false) + insecure: false + # HTTP configuration settings + http_config: {} + + deploymentMode: SimpleScalable + + backend: + replicas: 3 + read: + replicas: 3 + write: + replicas: 3 + + # Disable minio storage + minio: + enabled: false + + # Zero out replica counts of other deployment modes + singleBinary: + replicas: 0 + + ingester: + replicas: 0 + querier: + replicas: 0 + queryFrontend: + replicas: 0 + queryScheduler: + replicas: 0 + distributor: + replicas: 0 + compactor: + replicas: 0 + indexGateway: + replicas: 0 + bloomCompactor: + replicas: 0 + bloomGateway: + replicas: 0 +``` +```azure + loki: + schemaConfig: + configs: + - from: 2024-04-01 + store: tsdb + object_store: azure + schema: v13 + index: + prefix: loki_index_ + period: 24h + ingester: + chunk_encoding: snappy + tracing: + enabled: true + querier: + max_concurrent: 4 + + storage: + type: azure + azure: + # Name of the Azure Blob Storage account + accountName: + # Key associated with the Azure Blob Storage account + accountKey: + # Comprehensive connection string for Azure Blob Storage account (Can be used to replace endpoint, accountName, and accountKey) + connectionString: + # Flag indicating whether to use Azure Managed Identity for authentication + useManagedIdentity: false + # Flag indicating whether to use a federated token for authentication + useFederatedToken: false + # Client ID of the user-assigned managed identity (if applicable) + userAssignedId: + # Timeout duration for requests made to the Azure Blob Storage account (in seconds) + requestTimeout: + # Domain suffix of the Azure Blob Storage service endpoint (e.g., core.windows.net) + endpointSuffix: + bucketNames: + chunks: "chunks" + ruler: "ruler" + admin: "admin" deploymentMode: SimpleScalable @@ -197,6 +289,8 @@ To configure Loki to use AWS S3 as the object storage rather than MinIO, you nee replicas: 0 ``` +{{< /code >}} + To configure other storage providers, refer to the [Helm Chart Reference]({{< relref "../reference" >}}). ## Next Steps From ff2f925a0cc43e2bbd070a44a71bc6e6227fb92a Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Wed, 5 Jun 2024 15:48:05 +0100 Subject: [PATCH 12/12] fix: removed kind install page --- .../deploy-locally-with-kind.md | 171 ------------------ 1 file changed, 171 deletions(-) delete mode 100644 docs/sources/setup/install/helm/install-microservices/deploy-locally-with-kind.md diff --git a/docs/sources/setup/install/helm/install-microservices/deploy-locally-with-kind.md b/docs/sources/setup/install/helm/install-microservices/deploy-locally-with-kind.md deleted file mode 100644 index e064342b9cec..000000000000 --- a/docs/sources/setup/install/helm/install-microservices/deploy-locally-with-kind.md +++ /dev/null @@ -1,171 +0,0 @@ ---- -title: Deploy the Helm chart localy with Kind -menuTitle: Deploy the Helm chart localy with Kind -description: Installing Loki in microservice (distributed) with Kind for local development and testing purposes. -weight: 300 -keywords: ---- - - -# Deploy the Helm chart localy with Kind - -This guide walks you through testing the Grafana Loki Helm chart localy in microservice mode using [Kind](https://kind.sigs.k8s.io/), a tool for running local Kubernetes clusters using Docker container nodes. - -## Prerequisites - -To run through this tutorial, you need the following tools installed: -- [Docker](https://docs.docker.com/get-docker/) -- [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/) -- [Helm](https://helm.sh/docs/intro/install/) - -## Steps - -1. Create a Kind cluster config. Save the following configuration to a file named `kind-config.yaml`: - - ```yaml - kind: Cluster - apiVersion: kind.x-k8s.io/v1alpha4 - nodes: - - role: control-plane - - role: worker - - role: worker - ``` - -2. Create a Kind cluster using the configuration file: - - ```bash - kind create cluster --config kind-config.yaml - ``` - -3. Add the Grafana Helm chart repository to Helm: - - ```bash - helm repo add grafana https://grafana.github.io/helm-charts - ``` - -4. Update the chart repository: - - ```bash - helm repo update - ``` - -5. Create a `values.yaml` file to configure the Loki installation. The example below illustrates how to deploy Loki in test mode using MinIO as storage: - - ```yaml - loki: - schemaConfig: - configs: - - from: 2024-04-01 - store: tsdb - object_store: s3 - schema: v13 - index: - prefix: loki_index_ - period: 24h - ingester: - chunk_encoding: snappy - tracing: - enabled: true - querier: - # Default is 4, if you have enough memory and CPU you can increase, reduce if OOMing - max_concurrent: 4 - - #gateway: - # ingress: - # enabled: true - # hosts: - # - host: FIXME - # paths: - # - path: / - # pathType: Prefix - - deploymentMode: Distributed - - ingester: - replicas: 3 - querier: - replicas: 3 - maxUnavailable: 2 - queryFrontend: - replicas: 2 - maxUnavailable: 1 - queryScheduler: - replicas: 2 - distributor: - replicas: 3 - maxUnavailable: 2 - compactor: - replicas: 1 - indexGateway: - replicas: 2 - maxUnavailable: 1 - - bloomCompactor: - replicas: 0 - bloomGateway: - replicas: 0 - - # Enable minio for storage - minio: - enabled: true - - # Zero out replica counts of other deployment modes - backend: - replicas: 0 - read: - replicas: 0 - write: - replicas: 0 - - singleBinary: - replicas: 0 - ``` - -6. Install or upgrade the Loki deployment. - - To install: - ```bash - helm install --values values.yaml loki grafana/loki - ``` - - To upgrade: - ```bash - helm upgrade --values values.yaml loki grafana/loki - ``` - -7. Verify that Loki is running: - - ```bash - kubectl get pods - ``` - The output should an output similar to the following: - - ```bash - - loki-canary-8thrx 1/1 Running 0 167m - loki-canary-h965l 1/1 Running 0 167m - loki-canary-th8kb 1/1 Running 0 167m - loki-chunks-cache-0 0/2 Pending 0 167m - loki-compactor-0 1/1 Running 0 167m - loki-compactor-1 1/1 Running 0 167m - loki-distributor-7c9bb8f4dd-bcwc5 1/1 Running 0 167m - loki-distributor-7c9bb8f4dd-jh9h8 1/1 Running 0 167m - loki-distributor-7c9bb8f4dd-np5dw 1/1 Running 0 167m - loki-gateway-77bc447887-qgc56 1/1 Running 0 167m - loki-index-gateway-0 1/1 Running 0 167m - loki-index-gateway-1 1/1 Running 0 166m - loki-ingester-zone-a-0 1/1 Running 0 167m - loki-ingester-zone-b-0 1/1 Running 0 167m - loki-ingester-zone-c-0 1/1 Running 0 167m - loki-minio-0 1/1 Running 0 167m - loki-querier-bb8695c6d-bv9x2 1/1 Running 0 167m - loki-querier-bb8695c6d-bz2rw 1/1 Running 0 167m - loki-querier-bb8695c6d-z9qf8 1/1 Running 0 167m - loki-query-frontend-6659566b49-528j5 1/1 Running 0 167m - loki-query-frontend-6659566b49-84jtx 1/1 Running 0 167m - loki-query-frontend-6659566b49-9wfr7 1/1 Running 0 167m - loki-query-scheduler-f6dc4b949-fknfk 1/1 Running 0 167m - loki-query-scheduler-f6dc4b949-h4nwh 1/1 Running 0 167m - loki-query-scheduler-f6dc4b949-scfwp 1/1 Running 0 167m - loki-results-cache-0 2/2 Running 0 167m - - ``` -