diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 000000000000..d1bf7f1e82e4 --- /dev/null +++ b/.dockerignore @@ -0,0 +1,3 @@ +production/docker/.data +.cache +.git diff --git a/CHANGELOG.md b/CHANGELOG.md index 4efaada58f1e..e9eb2999b19a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,7 @@ #### Loki ##### Enhancements +* [6691](https://github.com/grafana/loki/pull/6691) **dannykopping**: Update production-ready Loki cluster in docker-compose * [6317](https://github.com/grafana/loki/pull/6317) **dannykoping**: General: add cache usage statistics * [6444](https://github.com/grafana/loki/pull/6444) **aminesnow** Add TLS config to query frontend. * [6372](https://github.com/grafana/loki/pull/6372) **splitice**: Add support for numbers in JSON fields. diff --git a/docs/sources/upgrading/_index.md b/docs/sources/upgrading/_index.md index 46e2634f6b16..e551adfda7a7 100644 --- a/docs/sources/upgrading/_index.md +++ b/docs/sources/upgrading/_index.md @@ -42,6 +42,16 @@ If you want to run at most a single querier per node, set `$._config.querier.use This value now defaults to 3100, so the Loki process doesn't require special privileges. Previously, it had been set to port 80, which is a privileged port. If you need Loki to listen on port 80, you can set it back to the previous default using `-server.http-listen-port=80`. +#### docker-compose setup has been updated + +The docker-compose [setup](https://github.com/grafana/loki/blob/main/production/docker) has been updated to **v2.6.0** and includes many improvements. + +Notable changes include: +- authentication (multi-tenancy) is **enabled** by default; you can disable it in `production/docker/config/loki.yaml` by setting `auth_enabled: false` +- storage is now using Minio instead of local filesystem + - move your current storage into `.data/minio` and it should work transparently +- log-generator was added - if you don't need it, simply remove the service from `docker-compose.yaml` or don't start the service + ## 2.6.0 ### Loki diff --git a/production/docker/.gitignore b/production/docker/.gitignore index 19e9d6513ad8..1443150cc2de 100644 --- a/production/docker/.gitignore +++ b/production/docker/.gitignore @@ -1 +1,2 @@ loki/ +.data diff --git a/production/docker/README.md b/production/docker/README.md index 0c522ce38d7a..750616d052bc 100644 --- a/production/docker/README.md +++ b/production/docker/README.md @@ -1,37 +1,84 @@ -# Loki cluster using docker-compose +# Loki with docker-compose -To deploy a cluster of loki nodes on a local machine (as shown below), you could use the `docker-compose-ha-member.yaml` file. +You can use this `docker-compose` setup to run Docker for development or in production. - +## Features -Some features of the deployment: +- Running in [Simple Scalable Deployment](https://grafana.com/docs/loki/latest/fundamentals/architecture/deployment-modes/#simple-scalable-deployment-mode) mode with 3 replicas for `read` and `write` targets +- Memberlist for [consistent hash](https://grafana.com/docs/loki/latest/fundamentals/architecture/rings/) ring +- [Minio](https://min.io/) for S3-compatible storage for chunks & indexes +- nginx gateway which acts as a reverse-proxy to the read/write paths +- Promtail for logs + - An optional log-generator +- Multi-tenancy enabled (`docker` as the tenant ID) +- Configuration for interactive debugging (see [Debugging](#debugging) section below) +- Prometheus for metric collection -- Backend: 3 Loki servers enabled with distributor, ingester, querier module -- Together they form a cluster ring based on memberlist mechanism (if using consul/etcd, modules can be separate for further separate read/write workloads) -- Index data are stored and replicated through botldb-shipper -- Replication_factor=2: the receiving distributor sends log data to 2 ingesters based on consistent hashing -- Chunk storage is a shared directory mounted from the same host directory (to simulate S3 or gcs) -- Query are performed through the two query frontend servers -- An nginx gateway to route the write and read workloads from clients (Grafana, promtail) +## Diagram -1. Ensure you have the most up-to-date Docker container images: +The below diagram describes the various components of this deployment, and how data flows between them. - ```bash - docker-compose pull - ``` +```mermaid +graph LR + Grafana --> |Query logs| nginx["nginx (port: 8080)"] + Promtail -->|Send logs| nginx -1. Run the stack on your local Docker: + nginx -.-> |read path| QueryFrontend["query-frontend"] + nginx -.-> |write path| Distributor - ```bash - docker-compose -f ./docker-compose-ha-memberlist.yaml up - ``` + QueryFrontend -.-> Querier -1. When adding data source in the grafana dashboard, using `http://loki-gateway:3100` for the URL field. + subgraph LokiRead["loki -target=read"] + Querier["querier"] + end -1. To clean up + subgraph Minio["Minio Storage"] + Chunks + Indexes + end - ```bash - docker-compose -f ./docker-compose-ha-memberlist.yaml down - ``` + subgraph LokiWrite["loki -target=write"] + Distributor["distributor"] -.-> Ingester["ingester"] + Ingester + end - Remove the data under `./loki`. + Querier --> |reads| Chunks & Indexes + Ingester --> |writes| Chunks & Indexes +``` + +## Getting Started + +Simply run `docker-compose up` and all the components will start. + +It'll take a few seconds for all the components to start up and register in the [ring](http://localhost:8080/ring). Once all instances are `ACTIVE`, Loki will start accepting reads and writes. All logs will be stored with the tenant ID `docker`. + +All data will be stored in the `.data` directory. + +The nginx gateway runs on port `8080` and you can access Loki through it. + +Prometheus runs on port `9090`, and you can access all metrics from Loki & Promtail here. + +Grafana runs on port `3000`, and there are Loki & Prometheus datasources enabled by default. + +## Endpoints + +- [`/ring`](http://localhost:8080/ring) - view all components registered in the hash ring +- [`/config`](http://localhost:8080/config) - view the configuration used by Loki +- [`/memberlist`](http://localhost:8080/memberlist) - view all components in the memberlist cluster +- [all other Loki API endpoints](https://grafana.com/docs/loki/latest/api/) + +## Debugging + +First, you'll need to build a Loki image that includes and runs [delve](https://github.com/go-delve/delve). + +Run `make loki-debug-image` from the root of this project. Grab the image name from the output (it'll look like `grafana/loki:...-debug`) and replace the Loki images in `docker-compose.yaml`. + +Next, view the `docker-compose.yaml` file and uncomment the sections related to debugging. + +You can follow [this guide](https://blog.jetbrains.com/go/2020/05/06/debugging-a-go-application-inside-a-docker-container/) to enable debugging in GoLand, but the basic steps are: + +1. Bind a host port to one of the Loki services +2. Add a _Go Remote_ debug configuration in GoLand and use that port +3. Run `docker-compose up` +4. Set a breakpoint and start the debug configuration +5. Build/debug something awesome :) \ No newline at end of file diff --git a/production/docker/config/datasources.yaml b/production/docker/config/datasources.yaml new file mode 100644 index 000000000000..cf6948fec545 --- /dev/null +++ b/production/docker/config/datasources.yaml @@ -0,0 +1,25 @@ +apiVersion: 1 +datasources: + - access: proxy + basicAuth: false + jsonData: + httpHeaderName1: "X-Scope-OrgID" + secureJsonData: + httpHeaderValue1: "docker" + editable: true + isDefault: true + name: loki + type: loki + uid: loki + url: http://loki-gateway + version: 1 + + - access: proxy + basicAuth: false + editable: true + isDefault: false + name: prometheus + type: prometheus + uid: prometheus + url: http://prometheus:9090 + version: 1 \ No newline at end of file diff --git a/production/docker/config/loki-docker-memberlist-boltdb-shipper.yaml b/production/docker/config/loki.yaml similarity index 62% rename from production/docker/config/loki-docker-memberlist-boltdb-shipper.yaml rename to production/docker/config/loki.yaml index c05675ef42c0..b3dd8d2551ec 100644 --- a/production/docker/config/loki-docker-memberlist-boltdb-shipper.yaml +++ b/production/docker/config/loki.yaml @@ -1,4 +1,4 @@ -auth_enabled: false +auth_enabled: true http_prefix: @@ -9,45 +9,50 @@ server: grpc_listen_port: 9095 log_level: info +common: + storage: + s3: + endpoint: minio:9000 + insecure: true + bucketnames: loki-data + access_key_id: loki + secret_access_key: supersecret + s3forcepathstyle: true + memberlist: - join_members: ["loki-1", "loki-2", "loki-3"] + join_members: ["loki-read", "loki-write"] dead_node_reclaim_time: 30s gossip_to_dead_nodes_time: 15s left_ingesters_timeout: 30s bind_addr: ['0.0.0.0'] bind_port: 7946 + gossip_interval: 2s ingester: lifecycler: - join_after: 60s + join_after: 10s observe_period: 5s ring: - replication_factor: 2 + replication_factor: 3 kvstore: store: memberlist final_sleep: 0s - chunk_idle_period: 1h + chunk_idle_period: 1m wal: enabled: true dir: /loki/wal - max_chunk_age: 1h + max_chunk_age: 1m chunk_retain_period: 30s chunk_encoding: snappy - chunk_target_size: 0 + chunk_target_size: 1.572864e+06 chunk_block_size: 262144 - # chunk_target_size: 1.572864e+06 - -# Only needed for global rate strategy -# distributor: -# ring: -# kvstore: -# store: memberlist + flush_op_timeout: 10s schema_config: configs: - from: 2020-08-01 store: boltdb-shipper - object_store: filesystem + object_store: s3 schema: v11 index: prefix: index_ @@ -55,18 +60,9 @@ schema_config: storage_config: boltdb_shipper: - # shared_store: s3 - shared_store: filesystem - active_index_directory: /loki/index - cache_location: /loki/boltdb-cache - - #aws: - # s3: s3://us-east-1/mybucket - # sse_encryption: true - # insecure: false - # s3forcepathstyle: true - filesystem: - directory: /loki/chunks + shared_store: s3 + active_index_directory: /tmp/index + cache_location: /tmp/boltdb-cache limits_config: @@ -93,21 +89,17 @@ query_range: parallelise_shardable_queries: true cache_results: true - results_cache: - cache: - # We're going to use the in-process "FIFO" cache - enable_fifocache: true - fifocache: - size: 1024 - validity: 24h - frontend: log_queries_longer_than: 5s compress_responses: true + max_outstanding_per_tenant: 2048 + +query_scheduler: + max_outstanding_requests_per_tenant: 1024 querier: query_ingesters_within: 2h compactor: - working_directory: /loki/compactor - shared_store: filesystem + working_directory: /tmp/compactor + shared_store: s3 diff --git a/production/docker/config/nginx-loki-gateway.conf b/production/docker/config/nginx.conf similarity index 57% rename from production/docker/config/nginx-loki-gateway.conf rename to production/docker/config/nginx.conf index 5df6822a20cc..45676d11c2e2 100644 --- a/production/docker/config/nginx-loki-gateway.conf +++ b/production/docker/config/nginx.conf @@ -7,7 +7,6 @@ events { } http { - default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] $status ' '"$request" $body_bytes_sent "$http_referer" ' @@ -16,16 +15,17 @@ http { sendfile on; tcp_nopush on; - upstream distributor { - server loki-1:3100; - server loki-2:3100; - server loki-3:3100; + upstream read { + server loki-read:3100; + } + + upstream write { + server loki-write:3100; } - upstream querier { - server loki-1:3100; - server loki-2:3100; - server loki-3:3100; + upstream cluster { + server loki-read:3100; + server loki-write:3100; } upstream query-frontend { @@ -34,18 +34,34 @@ http { server { listen 80; - proxy_set_header X-Scope-OrgID docker-ha; + listen 3100; - location = /loki/api/v1/push { - proxy_pass http://distributor$request_uri; - } - location = /ring { - proxy_pass http://distributor$request_uri; + proxy_pass http://cluster$request_uri; + } + + location = /memberlist { + proxy_pass http://cluster$request_uri; + } + + location = /config { + proxy_pass http://cluster$request_uri; + } + + location = /metrics { + proxy_pass http://cluster$request_uri; + } + + location = /ready { + proxy_pass http://cluster$request_uri; + } + + location = /loki/api/v1/push { + proxy_pass http://write$request_uri; } location = /loki/api/v1/tail { - proxy_pass http://querier$request_uri; + proxy_pass http://read$request_uri; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; } @@ -54,14 +70,4 @@ http { proxy_pass http://query-frontend$request_uri; } } - - server { - listen 3100; - proxy_set_header X-Scope-OrgID docker-ha; - - location ~ /loki/api/.* { - proxy_pass http://querier$request_uri; - } - - } } diff --git a/production/docker/config/prometheus.yaml b/production/docker/config/prometheus.yaml new file mode 100644 index 000000000000..ac6e57af20c3 --- /dev/null +++ b/production/docker/config/prometheus.yaml @@ -0,0 +1,22 @@ +global: + scrape_interval: 5s + +scrape_configs: + - job_name: 'prometheus' + static_configs: + - targets: + - 'prometheus:9090' + - job_name: 'loki' + dns_sd_configs: + - names: + - loki-read + - loki-write + - loki-frontend + type: A + port: 3100 + - job_name: 'promtail' + dns_sd_configs: + - names: + - promtail + type: A + port: 9080 \ No newline at end of file diff --git a/production/docker/config/promtail-gateway.yaml b/production/docker/config/promtail-gateway.yaml deleted file mode 100644 index 9ce3d2652fc5..000000000000 --- a/production/docker/config/promtail-gateway.yaml +++ /dev/null @@ -1,19 +0,0 @@ -server: - http_listen_port: 9080 - grpc_listen_port: 0 - log_level: "debug" - -positions: - filename: /tmp/positions.yaml - -clients: - - url: http://loki-gateway:80/loki/api/v1/push - -scrape_configs: - - job_name: system - static_configs: - - targets: - - localhost - labels: - job: varlogs - __path__: /var/log/*log diff --git a/production/docker/config/promtail.yaml b/production/docker/config/promtail.yaml new file mode 100644 index 000000000000..cfee3f2c4d1c --- /dev/null +++ b/production/docker/config/promtail.yaml @@ -0,0 +1,28 @@ +server: + http_listen_port: 9080 + grpc_listen_port: 0 + log_level: "info" + +positions: + filename: /tmp/positions.yaml + +clients: + - url: http://loki-gateway:80/loki/api/v1/push + tenant_id: docker + +scrape_configs: + - job_name: generated-logs + static_configs: + - targets: + - localhost + labels: + job: generated-logs + __path__: /var/log/generated-logs.txt + pipeline_stages: + - json: + expressions: + http_method: 'method' + http_status: "status" + - labels: + http_method: + http_status: \ No newline at end of file diff --git a/production/docker/docker-compose-ha-diagram.png b/production/docker/docker-compose-ha-diagram.png deleted file mode 100644 index 5033674be92f..000000000000 Binary files a/production/docker/docker-compose-ha-diagram.png and /dev/null differ diff --git a/production/docker/docker-compose-ha-memberlist.yaml b/production/docker/docker-compose-ha-memberlist.yaml deleted file mode 100644 index 782c1f73c835..000000000000 --- a/production/docker/docker-compose-ha-memberlist.yaml +++ /dev/null @@ -1,103 +0,0 @@ -version: "3.8" - -networks: - loki: - -services: - - # Since the Loki containers are running as user 10001 and the mounted data volume is owned by root, - # Loki would not have permissions to create the directories. - # Therefore the init container changes permissions of the mounted directory. - init: - image: grafana/loki:2.5.0 - user: root - entrypoint: - - "chown" - - "10001:10001" - - "/loki" - volumes: - - ./loki:/loki - networks: - - loki - - grafana: - image: grafana/grafana:8.3.3 - ports: - - "3000:3000" - networks: - - loki - - promtail: - image: grafana/promtail:2.5.0 - volumes: - - /var/log:/var/log - - ./config:/etc/promtail/ - ports: - - "9080:9080" - command: -config.file=/etc/promtail/promtail-gateway.yaml - networks: - - loki - - loki-gateway: - image: nginx:1.19 - volumes: - - ./config/nginx-loki-gateway.conf:/etc/nginx/nginx.conf - ports: - - "80" - - "3100" - networks: - - loki - - loki-frontend: - image: grafana/loki:2.5.0 - volumes: - - ./config:/etc/loki/ - ports: - - "3100" - command: "-config.file=/etc/loki/loki-docker-memberlist-boltdb-shipper.yaml -target=query-frontend -frontend.downstream-url=http://loki-gateway:3100" - networks: - - loki - deploy: - mode: replicated - replicas: 2 - - loki-1: - image: grafana/loki:2.5.0 - volumes: - - ./config:/etc/loki/ - - ./loki:/loki - ports: - - "3100" - - "7946" - command: "-config.file=/etc/loki/loki-docker-memberlist-boltdb-shipper.yaml -target=all" - networks: - - loki - restart: on-failure - - loki-2: - image: grafana/loki:2.5.0 - volumes: - - ./config:/etc/loki/ - - ./loki:/loki - ports: - - "3100" - - "7946" - command: "-config.file=/etc/loki/loki-docker-memberlist-boltdb-shipper.yaml -target=all" - # command: "-config.file=/etc/loki/loki-config.yaml" - networks: - - loki - restart: on-failure - - loki-3: - image: grafana/loki:2.5.0 - volumes: - - ./config:/etc/loki/ - - ./loki:/loki - ports: - - "3100" - - "7946" - command: "-config.file=/etc/loki/loki-docker-memberlist-boltdb-shipper.yaml -target=all" - # command: "-config.file=/etc/loki/loki-config.yaml" - networks: - - loki - restart: on-failure diff --git a/production/docker/docker-compose.yaml b/production/docker/docker-compose.yaml new file mode 100644 index 000000000000..1611f5a0bd6a --- /dev/null +++ b/production/docker/docker-compose.yaml @@ -0,0 +1,164 @@ +version: "3.8" + +networks: + loki: + +services: + + # Since the Loki containers are running as user 10001 and the mounted data volume is owned by root, + # Loki would not have permissions to create the directories. + # Therefore the init container changes permissions of the mounted directory. + init: + image: grafana/loki:2.6.0 + user: root + entrypoint: + - "chown" + - "10001:10001" + - "/loki" + volumes: + - ./loki:/loki + networks: + - loki + + grafana: + image: grafana/grafana:8.5.9 + ports: + - "3000:3000" + environment: + GF_AUTH_ANONYMOUS_ENABLED: "true" + GF_AUTH_DISABLE_LOGIN_FORM: "true" + GF_AUTH_ANONYMOUS_ORG_ROLE: "Admin" + volumes: + - ./config/datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yml + - grafana:/var/lib/grafana + networks: + - loki + + prometheus: + image: prom/prometheus:v2.27.0 + ports: + - 9090:9090 + volumes: + - ./config/prometheus.yaml:/etc/prometheus/prometheus.yml + - prometheus:/prometheus + command: [ '--log.level=debug', '--config.file=/etc/prometheus/prometheus.yml', '--query.lookback-delta=30s' ] + networks: + - loki + + # for testing purposes only, disable in production + log-generator: + image: mingrammer/flog + command: + - --loop + - --format=json + - --number=10 # number of log lines to generate per second + - --delay=100ms # delay between log lines + - --output=/var/log/generated-logs.txt + - --overwrite + - --type=log + volumes: + - ./loki/:/var/log/ + + promtail: + image: grafana/promtail:2.6.0 + volumes: + - ./loki/:/var/log/ + - ./config:/etc/promtail/ + ports: + - "9080:9080" + command: -config.file=/etc/promtail/promtail.yaml + networks: + - loki + + minio: + image: minio/minio + entrypoint: + - sh + - -euc + - | + mkdir -p /data/loki-data && \ + mkdir -p /data/loki-ruler && + minio server /data + environment: + - MINIO_ACCESS_KEY=loki + - MINIO_SECRET_KEY=supersecret + - MINIO_PROMETHEUS_AUTH_TYPE=public + - MINIO_UPDATE=off + ports: + - "9000:9000" + volumes: + - ./.data/minio:/data + networks: + - loki + + loki-gateway: + image: nginx:1.19 + volumes: + - ./config/nginx.conf:/etc/nginx/nginx.conf + ports: + - "8080:80" + - "3100" + networks: + - loki + + loki-frontend: + image: grafana/loki:2.6.0 + volumes: + - ./config:/etc/loki/ + ports: + - "3100" + command: "-config.file=/etc/loki/loki.yaml -target=query-frontend -frontend.downstream-url=http://loki-read:3100" + networks: + - loki + deploy: + mode: replicated + replicas: 2 + + loki-read: + image: grafana/loki:2.6.0 + volumes: + - ./config:/etc/loki/ + ports: + - "3100" + - "7946" + + # uncomment to use interactive debugging + # - "40000-40002:40000" # # makes the replicas available on ports 40000, 40001, 40002 + #cap_add: + # - SYS_PTRACE + #security_opt: + # - apparmor=unconfined + command: "-config.file=/etc/loki/loki.yaml -target=read" + networks: + - loki + restart: always + deploy: + mode: replicated + replicas: 3 + # only needed for interactive debugging with dlv + + loki-write: + image: grafana/loki:2.6.0 + volumes: + - ./config:/etc/loki/ + ports: + - "3100" + - "7946" + + # uncomment to use interactive debugging + # - "50000-50002:40000" # makes the replicas available on ports 50000, 50001, 50002 + # cap_add: + # - SYS_PTRACE + # security_opt: + # - apparmor=unconfined + command: "-config.file=/etc/loki/loki.yaml -target=write" + networks: + - loki + restart: always + deploy: + mode: replicated + replicas: 3 + +volumes: + prometheus: + grafana: \ No newline at end of file