From 9b1462f1b15f3a56d4d51ac471f0311f17b299f6 Mon Sep 17 00:00:00 2001 From: Madhav Jivrajani Date: Mon, 5 Oct 2020 23:34:25 +0530 Subject: [PATCH] [DOCS] change format of unordered lists in tech docs --- docs/sources/architecture/_index.md | 34 ++-- docs/sources/clients/aws/_index.md | 6 +- docs/sources/clients/fluentbit/_index.md | 4 +- docs/sources/clients/promtail/_index.md | 6 +- .../sources/clients/promtail/configuration.md | 160 +++++++++--------- docs/sources/clients/promtail/pipelines.md | 24 +-- docs/sources/clients/promtail/scraping.md | 20 +-- .../sources/clients/promtail/stages/_index.md | 28 +-- docs/sources/community/governance.md | 30 ++-- docs/sources/configuration/_index.md | 18 +- docs/sources/design-documents/labels.md | 34 ++-- docs/sources/operations/loki-canary.md | 6 +- docs/sources/operations/storage/_index.md | 74 ++++---- docs/sources/operations/upgrade.md | 8 +- docs/sources/overview/_index.md | 12 +- 15 files changed, 232 insertions(+), 232 deletions(-) diff --git a/docs/sources/architecture/_index.md b/docs/sources/architecture/_index.md index 8a3b67a9dbbb..ce8683679428 100644 --- a/docs/sources/architecture/_index.md +++ b/docs/sources/architecture/_index.md @@ -186,9 +186,9 @@ Query frontends are **stateless**. However, due to how the internal queue works, The query frontend queuing mechanism is used to: -* Ensure that large queries, that could cause an out-of-memory (OOM) error in the querier, will be retried on failure. This allows administrators to under-provision memory for queries, or optimistically run more small queries in parallel, which helps to reduce the TCO. -* Prevent multiple large requests from being convoyed on a single querier by distributing them across all queriers using a first-in/first-out queue (FIFO). -* Prevent a single tenant from denial-of-service-ing (DOSing) other tenants by fairly scheduling queries between tenants. +- Ensure that large queries, that could cause an out-of-memory (OOM) error in the querier, will be retried on failure. This allows administrators to under-provision memory for queries, or optimistically run more small queries in parallel, which helps to reduce the TCO. +- Prevent multiple large requests from being convoyed on a single querier by distributing them across all queriers using a first-in/first-out queue (FIFO). +- Prevent a single tenant from denial-of-service-ing (DOSing) other tenants by fairly scheduling queries between tenants. #### Splitting @@ -277,16 +277,16 @@ The **chunk store** is Loki's long-term data store, designed to support interactive querying and sustained writing without the need for background maintenance tasks. It consists of: -* An index for the chunks. This index can be backed by: - * [Amazon DynamoDB](https://aws.amazon.com/dynamodb) - * [Google Bigtable](https://cloud.google.com/bigtable) - * [Apache Cassandra](https://cassandra.apache.org) -* A key-value (KV) store for the chunk data itself, which can be: - * [Amazon DynamoDB](https://aws.amazon.com/dynamodb) - * [Google Bigtable](https://cloud.google.com/bigtable) - * [Apache Cassandra](https://cassandra.apache.org) - * [Amazon S3](https://aws.amazon.com/s3) - * [Google Cloud Storage](https://cloud.google.com/storage/) +- An index for the chunks. This index can be backed by: + - [Amazon DynamoDB](https://aws.amazon.com/dynamodb) + - [Google Bigtable](https://cloud.google.com/bigtable) + - [Apache Cassandra](https://cassandra.apache.org) +- A key-value (KV) store for the chunk data itself, which can be: + - [Amazon DynamoDB](https://aws.amazon.com/dynamodb) + - [Google Bigtable](https://cloud.google.com/bigtable) + - [Apache Cassandra](https://cassandra.apache.org) + - [Amazon S3](https://aws.amazon.com/s3) + - [Google Cloud Storage](https://cloud.google.com/storage/) > Unlike the other core components of Loki, the chunk store is not a separate > service, job, or process, but rather a library embedded in the two services @@ -297,16 +297,16 @@ The chunk store relies on a unified interface to the Cassandra) that can be used to back the chunk store index. This interface assumes that the index is a collection of entries keyed by: -* A **hash key**. This is required for *all* reads and writes. -* A **range key**. This is required for writes and can be omitted for reads, +- A **hash key**. This is required for *all* reads and writes. +- A **range key**. This is required for writes and can be omitted for reads, which can be queried by prefix or range. The interface works somewhat differently across the supported databases: -* DynamoDB supports range and hash keys natively. Index entries are thus +- DynamoDB supports range and hash keys natively. Index entries are thus modelled directly as DynamoDB entries, with the hash key as the distribution key and the range as the DynamoDB range key. -* For Bigtable and Cassandra, index entries are modelled as individual column +- For Bigtable and Cassandra, index entries are modelled as individual column values. The hash key becomes the row key and the range key becomes the column key. diff --git a/docs/sources/clients/aws/_index.md b/docs/sources/clients/aws/_index.md index 30eaf4521f60..fd444126d811 100644 --- a/docs/sources/clients/aws/_index.md +++ b/docs/sources/clients/aws/_index.md @@ -4,6 +4,6 @@ title: AWS Sending logs from AWS services to Loki is a little different depending on what AWS service you are using: -* [EC2](ec2/) -* [ECS](ecs/) -* [EKS](eks/) \ No newline at end of file +- [EC2](ec2/) +- [ECS](ecs/) +- [EKS](eks/) \ No newline at end of file diff --git a/docs/sources/clients/fluentbit/_index.md b/docs/sources/clients/fluentbit/_index.md index 255ffd5821ae..efe8cf529710 100644 --- a/docs/sources/clients/fluentbit/_index.md +++ b/docs/sources/clients/fluentbit/_index.md @@ -146,8 +146,8 @@ If you don't want the `kubernetes` and `HOSTNAME` fields to appear in the log li Buffering refers to the ability to store the records somewhere, and while they are processed and delivered, still be able to store more. Loki output plugin in certain situation can be blocked by loki client because of its design: -* BatchSize is over limit, output plugin pause receiving new records until the pending batch is successfully sent to the server -* Loki server is unreachable (retry 429s, 500s and connection-level errors), output plugin blocks new records until loki server will be available again and the pending batch is successfully sent to the server or as long as the maximum number of attempts has been reached within configured back-off mechanism +- BatchSize is over limit, output plugin pause receiving new records until the pending batch is successfully sent to the server +- Loki server is unreachable (retry 429s, 500s and connection-level errors), output plugin blocks new records until loki server will be available again and the pending batch is successfully sent to the server or as long as the maximum number of attempts has been reached within configured back-off mechanism The blocking state with some of the input plugins is not acceptable because it can have a undesirable side effects on the part that generates the logs. Fluent Bit implements buffering mechanism that is based on parallel processing and it cannot send logs in order which is loki requirement (loki logs must be in increasing time order per stream). diff --git a/docs/sources/clients/promtail/_index.md b/docs/sources/clients/promtail/_index.md index b541c419754c..3b3fcf6d3555 100644 --- a/docs/sources/clients/promtail/_index.md +++ b/docs/sources/clients/promtail/_index.md @@ -41,9 +41,9 @@ Promtail can also be configured to receive logs from another Promtail or any Lok There are a few instances where this might be helpful: -* complex network infrastructures where many machines having egress is not desirable. -* using the Docker Logging Driver and wanting to provide a complex pipeline or to extract metrics from logs. -* serverless setups where many ephemeral log sources want to send to Loki, sending to a Promtail instance with `use_incoming_timestamp` == false can avoid out of order errors and avoid having to use high cardinality labels. +- complex network infrastructures where many machines having egress is not desirable. +- using the Docker Logging Driver and wanting to provide a complex pipeline or to extract metrics from logs. +- serverless setups where many ephemeral log sources want to send to Loki, sending to a Promtail instance with `use_incoming_timestamp` == false can avoid out of order errors and avoid having to use high cardinality labels. ## Receiving logs From Syslog diff --git a/docs/sources/clients/promtail/configuration.md b/docs/sources/clients/promtail/configuration.md index 012bbcec80c1..77db37c970f6 100644 --- a/docs/sources/clients/promtail/configuration.md +++ b/docs/sources/clients/promtail/configuration.md @@ -82,16 +82,16 @@ from scraped targets, see [Pipelines](../pipelines/). Generic placeholders are defined as follows: -* ``: a boolean that can take the values `true` or `false` -* ``: any integer matching the regular expression `[1-9]+[0-9]*` -* ``: a duration matching the regular expression `[0-9]+(ms|[smhdwy])` -* ``: a string matching the regular expression `[a-zA-Z_][a-zA-Z0-9_]*` -* ``: a string of Unicode characters -* ``: a valid path relative to current working directory or an +- ``: a boolean that can take the values `true` or `false` +- ``: any integer matching the regular expression `[1-9]+[0-9]*` +- ``: a duration matching the regular expression `[0-9]+(ms|[smhdwy])` +- ``: a string matching the regular expression `[a-zA-Z_][a-zA-Z0-9_]*` +- ``: a string of Unicode characters +- ``: a valid path relative to current working directory or an absolute path. -* ``: a valid string consisting of a hostname or IP followed by an optional port number -* ``: a regular string -* ``: a regular string that is a secret, such as a password +- ``: a valid string consisting of a hostname or IP followed by an optional port number +- ``: a regular string +- ``: a regular string that is a secret, such as a password Supported contents and default values of `config.yaml`: @@ -752,15 +752,15 @@ labels: #### Available Labels -* `__syslog_connection_ip_address`: The remote IP address. -* `__syslog_connection_hostname`: The remote hostname. -* `__syslog_message_severity`: The [syslog severity](https://tools.ietf.org/html/rfc5424#section-6.2.1) parsed from the message. Symbolic name as per [syslog_message.go](https://github.com/influxdata/go-syslog/blob/v2.0.1/rfc5424/syslog_message.go#L184). -* `__syslog_message_facility`: The [syslog facility](https://tools.ietf.org/html/rfc5424#section-6.2.1) parsed from the message. Symbolic name as per [syslog_message.go](https://github.com/influxdata/go-syslog/blob/v2.0.1/rfc5424/syslog_message.go#L235) and `syslog(3)`. -* `__syslog_message_hostname`: The [hostname](https://tools.ietf.org/html/rfc5424#section-6.2.4) parsed from the message. -* `__syslog_message_app_name`: The [app-name field](https://tools.ietf.org/html/rfc5424#section-6.2.5) parsed from the message. -* `__syslog_message_proc_id`: The [procid field](https://tools.ietf.org/html/rfc5424#section-6.2.6) parsed from the message. -* `__syslog_message_msg_id`: The [msgid field](https://tools.ietf.org/html/rfc5424#section-6.2.7) parsed from the message. -* `__syslog_message_sd_[_]_`: The [structured-data field](https://tools.ietf.org/html/rfc5424#section-6.3) parsed from the message. The data field `[custom@99770 example="1"]` becomes `__syslog_message_sd_custom_99770_example`. +- `__syslog_connection_ip_address`: The remote IP address. +- `__syslog_connection_hostname`: The remote hostname. +- `__syslog_message_severity`: The [syslog severity](https://tools.ietf.org/html/rfc5424#section-6.2.1) parsed from the message. Symbolic name as per [syslog_message.go](https://github.com/influxdata/go-syslog/blob/v2.0.1/rfc5424/syslog_message.go#L184). +- `__syslog_message_facility`: The [syslog facility](https://tools.ietf.org/html/rfc5424#section-6.2.1) parsed from the message. Symbolic name as per [syslog_message.go](https://github.com/influxdata/go-syslog/blob/v2.0.1/rfc5424/syslog_message.go#L235) and `syslog(3)`. +- `__syslog_message_hostname`: The [hostname](https://tools.ietf.org/html/rfc5424#section-6.2.4) parsed from the message. +- `__syslog_message_app_name`: The [app-name field](https://tools.ietf.org/html/rfc5424#section-6.2.5) parsed from the message. +- `__syslog_message_proc_id`: The [procid field](https://tools.ietf.org/html/rfc5424#section-6.2.6) parsed from the message. +- `__syslog_message_msg_id`: The [msgid field](https://tools.ietf.org/html/rfc5424#section-6.2.7) parsed from the message. +- `__syslog_message_sd_[_]_`: The [structured-data field](https://tools.ietf.org/html/rfc5424#section-6.3) parsed from the message. The data field `[custom@99770 example="1"]` becomes `__syslog_message_sd_custom_99770_example`. ### loki_push_api_config @@ -846,19 +846,19 @@ use `.*.*`. `` determines the relabeling action to take: -* `replace`: Match `regex` against the concatenated `source_labels`. Then, set +- `replace`: Match `regex` against the concatenated `source_labels`. Then, set `target_label` to `replacement`, with match group references (`${1}`, `${2}`, ...) in `replacement` substituted by their value. If `regex` does not match, no replacement takes place. -* `keep`: Drop targets for which `regex` does not match the concatenated `source_labels`. -* `drop`: Drop targets for which `regex` matches the concatenated `source_labels`. -* `hashmod`: Set `target_label` to the `modulus` of a hash of the concatenated `source_labels`. -* `labelmap`: Match `regex` against all label names. Then copy the values of the matching labels +- `keep`: Drop targets for which `regex` does not match the concatenated `source_labels`. +- `drop`: Drop targets for which `regex` matches the concatenated `source_labels`. +- `hashmod`: Set `target_label` to the `modulus` of a hash of the concatenated `source_labels`. +- `labelmap`: Match `regex` against all label names. Then copy the values of the matching labels to label names given by `replacement` with match group references (`${1}`, `${2}`, ...) in `replacement` substituted by their value. -* `labeldrop`: Match `regex` against all label names. Any label that matches will be +- `labeldrop`: Match `regex` against all label names. Any label that matches will be removed from the set of labels. -* `labelkeep`: Match `regex` against all label names. Any label that does not match will be +- `labelkeep`: Match `regex` against all label names. Any label that does not match will be removed from the set of labels. Care must be taken with `labeldrop` and `labelkeep` to ensure that logs are @@ -954,12 +954,12 @@ node object in the address type order of `NodeInternalIP`, `NodeExternalIP`, Available meta labels: -* `__meta_kubernetes_node_name`: The name of the node object. -* `__meta_kubernetes_node_label_`: Each label from the node object. -* `__meta_kubernetes_node_labelpresent_`: `true` for each label from the node object. -* `__meta_kubernetes_node_annotation_`: Each annotation from the node object. -* `__meta_kubernetes_node_annotationpresent_`: `true` for each annotation from the node object. -* `__meta_kubernetes_node_address_`: The first address for each node address type, if it exists. +- `__meta_kubernetes_node_name`: The name of the node object. +- `__meta_kubernetes_node_label_`: Each label from the node object. +- `__meta_kubernetes_node_labelpresent_`: `true` for each label from the node object. +- `__meta_kubernetes_node_annotation_`: Each annotation from the node object. +- `__meta_kubernetes_node_annotationpresent_`: `true` for each annotation from the node object. +- `__meta_kubernetes_node_address_`: The first address for each node address type, if it exists. In addition, the `instance` label for the node will be set to the node name as retrieved from the API server. @@ -973,16 +973,16 @@ service port. Available meta labels: -* `__meta_kubernetes_namespace`: The namespace of the service object. -* `__meta_kubernetes_service_annotation_`: Each annotation from the service object. -* `__meta_kubernetes_service_annotationpresent_`: "true" for each annotation of the service object. -* `__meta_kubernetes_service_cluster_ip`: The cluster IP address of the service. (Does not apply to services of type ExternalName) -* `__meta_kubernetes_service_external_name`: The DNS name of the service. (Applies to services of type ExternalName) -* `__meta_kubernetes_service_label_`: Each label from the service object. -* `__meta_kubernetes_service_labelpresent_`: `true` for each label of the service object. -* `__meta_kubernetes_service_name`: The name of the service object. -* `__meta_kubernetes_service_port_name`: Name of the service port for the target. -* `__meta_kubernetes_service_port_protocol`: Protocol of the service port for the target. +- `__meta_kubernetes_namespace`: The namespace of the service object. +- `__meta_kubernetes_service_annotation_`: Each annotation from the service object. +- `__meta_kubernetes_service_annotationpresent_`: "true" for each annotation of the service object. +- `__meta_kubernetes_service_cluster_ip`: The cluster IP address of the service. (Does not apply to services of type ExternalName) +- `__meta_kubernetes_service_external_name`: The DNS name of the service. (Applies to services of type ExternalName) +- `__meta_kubernetes_service_label_`: Each label from the service object. +- `__meta_kubernetes_service_labelpresent_`: `true` for each label of the service object. +- `__meta_kubernetes_service_name`: The name of the service object. +- `__meta_kubernetes_service_port_name`: Name of the service port for the target. +- `__meta_kubernetes_service_port_protocol`: Protocol of the service port for the target. #### `pod` @@ -993,26 +993,26 @@ adding a port via relabeling. Available meta labels: -* `__meta_kubernetes_namespace`: The namespace of the pod object. -* `__meta_kubernetes_pod_name`: The name of the pod object. -* `__meta_kubernetes_pod_ip`: The pod IP of the pod object. -* `__meta_kubernetes_pod_label_`: Each label from the pod object. -* `__meta_kubernetes_pod_labelpresent_`: `true`for each label from the pod object. -* `__meta_kubernetes_pod_annotation_`: Each annotation from the pod object. -* `__meta_kubernetes_pod_annotationpresent_`: `true` for each annotation from the pod object. -* `__meta_kubernetes_pod_container_init`: `true` if the container is an [InitContainer](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/) -* `__meta_kubernetes_pod_container_name`: Name of the container the target address points to. -* `__meta_kubernetes_pod_container_port_name`: Name of the container port. -* `__meta_kubernetes_pod_container_port_number`: Number of the container port. -* `__meta_kubernetes_pod_container_port_protocol`: Protocol of the container port. -* `__meta_kubernetes_pod_ready`: Set to `true` or `false` for the pod's ready state. -* `__meta_kubernetes_pod_phase`: Set to `Pending`, `Running`, `Succeeded`, `Failed` or `Unknown` +- `__meta_kubernetes_namespace`: The namespace of the pod object. +- `__meta_kubernetes_pod_name`: The name of the pod object. +- `__meta_kubernetes_pod_ip`: The pod IP of the pod object. +- `__meta_kubernetes_pod_label_`: Each label from the pod object. +- `__meta_kubernetes_pod_labelpresent_`: `true`for each label from the pod object. +- `__meta_kubernetes_pod_annotation_`: Each annotation from the pod object. +- `__meta_kubernetes_pod_annotationpresent_`: `true` for each annotation from the pod object. +- `__meta_kubernetes_pod_container_init`: `true` if the container is an [InitContainer](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/) +- `__meta_kubernetes_pod_container_name`: Name of the container the target address points to. +- `__meta_kubernetes_pod_container_port_name`: Name of the container port. +- `__meta_kubernetes_pod_container_port_number`: Number of the container port. +- `__meta_kubernetes_pod_container_port_protocol`: Protocol of the container port. +- `__meta_kubernetes_pod_ready`: Set to `true` or `false` for the pod's ready state. +- `__meta_kubernetes_pod_phase`: Set to `Pending`, `Running`, `Succeeded`, `Failed` or `Unknown` in the [lifecycle](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase). -* `__meta_kubernetes_pod_node_name`: The name of the node the pod is scheduled onto. -* `__meta_kubernetes_pod_host_ip`: The current host IP of the pod object. -* `__meta_kubernetes_pod_uid`: The UID of the pod object. -* `__meta_kubernetes_pod_controller_kind`: Object kind of the pod controller. -* `__meta_kubernetes_pod_controller_name`: Name of the pod controller. +- `__meta_kubernetes_pod_node_name`: The name of the node the pod is scheduled onto. +- `__meta_kubernetes_pod_host_ip`: The current host IP of the pod object. +- `__meta_kubernetes_pod_uid`: The UID of the pod object. +- `__meta_kubernetes_pod_controller_kind`: Object kind of the pod controller. +- `__meta_kubernetes_pod_controller_name`: Name of the pod controller. #### `endpoints` @@ -1023,19 +1023,19 @@ endpoint port, are discovered as targets as well. Available meta labels: -* `__meta_kubernetes_namespace`: The namespace of the endpoints object. -* `__meta_kubernetes_endpoints_name`: The names of the endpoints object. -* For all targets discovered directly from the endpoints list (those not additionally inferred +- `__meta_kubernetes_namespace`: The namespace of the endpoints object. +- `__meta_kubernetes_endpoints_name`: The names of the endpoints object. +- For all targets discovered directly from the endpoints list (those not additionally inferred from underlying pods), the following labels are attached: - * `__meta_kubernetes_endpoint_hostname`: Hostname of the endpoint. - * `__meta_kubernetes_endpoint_node_name`: Name of the node hosting the endpoint. - * `__meta_kubernetes_endpoint_ready`: Set to `true` or `false` for the endpoint's ready state. - * `__meta_kubernetes_endpoint_port_name`: Name of the endpoint port. - * `__meta_kubernetes_endpoint_port_protocol`: Protocol of the endpoint port. - * `__meta_kubernetes_endpoint_address_target_kind`: Kind of the endpoint address target. - * `__meta_kubernetes_endpoint_address_target_name`: Name of the endpoint address target. -* If the endpoints belong to a service, all labels of the `role: service` discovery are attached. -* For all targets backed by a pod, all labels of the `role: pod` discovery are attached. + - `__meta_kubernetes_endpoint_hostname`: Hostname of the endpoint. + - `__meta_kubernetes_endpoint_node_name`: Name of the node hosting the endpoint. + - `__meta_kubernetes_endpoint_ready`: Set to `true` or `false` for the endpoint's ready state. + - `__meta_kubernetes_endpoint_port_name`: Name of the endpoint port. + - `__meta_kubernetes_endpoint_port_protocol`: Protocol of the endpoint port. + - `__meta_kubernetes_endpoint_address_target_kind`: Kind of the endpoint address target. + - `__meta_kubernetes_endpoint_address_target_name`: Name of the endpoint address target. +- If the endpoints belong to a service, all labels of the `role: service` discovery are attached. +- For all targets backed by a pod, all labels of the `role: pod` discovery are attached. #### `ingress` @@ -1045,15 +1045,15 @@ The address will be set to the host specified in the ingress spec. Available meta labels: -* `__meta_kubernetes_namespace`: The namespace of the ingress object. -* `__meta_kubernetes_ingress_name`: The name of the ingress object. -* `__meta_kubernetes_ingress_label_`: Each label from the ingress object. -* `__meta_kubernetes_ingress_labelpresent_`: `true` for each label from the ingress object. -* `__meta_kubernetes_ingress_annotation_`: Each annotation from the ingress object. -* `__meta_kubernetes_ingress_annotationpresent_`: `true` for each annotation from the ingress object. -* `__meta_kubernetes_ingress_scheme`: Protocol scheme of ingress, `https` if TLS +- `__meta_kubernetes_namespace`: The namespace of the ingress object. +- `__meta_kubernetes_ingress_name`: The name of the ingress object. +- `__meta_kubernetes_ingress_label_`: Each label from the ingress object. +- `__meta_kubernetes_ingress_labelpresent_`: `true` for each label from the ingress object. +- `__meta_kubernetes_ingress_annotation_`: Each annotation from the ingress object. +- `__meta_kubernetes_ingress_annotationpresent_`: `true` for each annotation from the ingress object. +- `__meta_kubernetes_ingress_scheme`: Protocol scheme of ingress, `https` if TLS config is set. Defaults to `http`. -* `__meta_kubernetes_ingress_path`: Path from ingress spec. Defaults to `/`. +- `__meta_kubernetes_ingress_path`: Path from ingress spec. Defaults to `/`. See below for the configuration options for Kubernetes discovery: diff --git a/docs/sources/clients/promtail/pipelines.md b/docs/sources/clients/promtail/pipelines.md index b913ba82f085..8baa38f156c1 100644 --- a/docs/sources/clients/promtail/pipelines.md +++ b/docs/sources/clients/promtail/pipelines.md @@ -199,24 +199,24 @@ given log entry. Parsing stages: - * [docker](../stages/docker/): Extract data by parsing the log line using the standard Docker format. - * [cri](../stages/cri/): Extract data by parsing the log line using the standard CRI format. - * [regex](../stages/regex/): Extract data using a regular expression. - * [json](../stages/json/): Extract data by parsing the log line as JSON. + - [docker](../stages/docker/): Extract data by parsing the log line using the standard Docker format. + - [cri](../stages/cri/): Extract data by parsing the log line using the standard CRI format. + - [regex](../stages/regex/): Extract data using a regular expression. + - [json](../stages/json/): Extract data by parsing the log line as JSON. Transform stages: - * [template](../stages/template/): Use Go templates to modify extracted data. + - [template](../stages/template/): Use Go templates to modify extracted data. Action stages: - * [timestamp](../stages/timestamp/): Set the timestamp value for the log entry. - * [output](../stages/output/): Set the log line text. - * [labels](../stages/labels/): Update the label set for the log entry. - * [metrics](../stages/metrics/): Calculate metrics based on extracted data. - * [tenant](../stages/tenant/): Set the tenant ID value to use for the log entry. + - [timestamp](../stages/timestamp/): Set the timestamp value for the log entry. + - [output](../stages/output/): Set the log line text. + - [labels](../stages/labels/): Update the label set for the log entry. + - [metrics](../stages/metrics/): Calculate metrics based on extracted data. + - [tenant](../stages/tenant/): Set the tenant ID value to use for the log entry. Filtering stages: - * [match](../stages/match/): Conditionally run stages based on the label set. - * [drop](../stages/drop/): Conditionally drop log lines based on several options. + - [match](../stages/match/): Conditionally run stages based on the label set. + - [drop](../stages/drop/): Conditionally drop log lines based on several options. diff --git a/docs/sources/clients/promtail/scraping.md b/docs/sources/clients/promtail/scraping.md index 010dd1bd1826..dc4a75daddb5 100644 --- a/docs/sources/clients/promtail/scraping.md +++ b/docs/sources/clients/promtail/scraping.md @@ -30,30 +30,30 @@ different labels. There are different types of labels present in Promtail: -* Labels starting with `__` (two underscores) are internal labels. They usually +- Labels starting with `__` (two underscores) are internal labels. They usually come from dynamic sources like service discovery. Once relabeling is done, they are removed from the label set. To persist internal labels so they're sent to Loki, rename them so they don't start with `__`. See [Relabeling](#relabeling) for more information. -* Labels starting with `__meta_kubernetes_pod_label_*` are "meta labels" which +- Labels starting with `__meta_kubernetes_pod_label_*` are "meta labels" which are generated based on your Kubernetes pod's labels. For example, if your Kubernetes pod has a label `name` set to `foobar`, then the `scrape_configs` section will receive an internal label `__meta_kubernetes_pod_label_name` with a value set to `foobar`. -* Other labels starting with `__meta_kubernetes_*` exist based on other +- Other labels starting with `__meta_kubernetes_*` exist based on other Kubernetes metadata, such as the namespace of the pod (`__meta_kubernetes_namespace`) or the name of the container inside the pod (`__meta_kubernetes_pod_container_name`). Refer to [the Prometheus docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config) for the full list of Kubernetes meta labels. -* The `__path__` label is a special label which Promtail uses after discovery to +- The `__path__` label is a special label which Promtail uses after discovery to figure out where the file to read is located. Wildcards are allowed, for example `/var/log/*.log` to get all files with a `log` extension in the specified directory, and `/var/log/**/*.log` for matching files and directories recursively. For a full list of options check out the [docs for the library promtail uses.](https://github.com/bmatcuk/doublestar) -* The label `filename` is added for every file found in `__path__` to ensure the +- The label `filename` is added for every file found in `__path__` to ensure the uniqueness of the streams. It is set to the absolute path of the file the line was read from. @@ -229,14 +229,14 @@ value or transformed to a final external label, such as `__job__`. ### Examples -* Drop the target if a label (`__service__` in the example) is empty: +- Drop the target if a label (`__service__` in the example) is empty: ```yaml - action: drop regex: '' source_labels: - __service__ ``` -* Drop the target if any of the `source_labels` contain a value: +- Drop the target if any of the `source_labels` contain a value: ```yaml - action: drop regex: .+ @@ -245,14 +245,14 @@ value or transformed to a final external label, such as `__job__`. - __meta_kubernetes_pod_label_name - __meta_kubernetes_pod_label_app ``` -* Persist an internal label by renaming it so it will be sent to Loki: +- Persist an internal label by renaming it so it will be sent to Loki: ```yaml - action: replace source_labels: - __meta_kubernetes_namespace target_label: namespace ``` -* Persist all Kubernetes pod labels by mapping them, like by mapping +- Persist all Kubernetes pod labels by mapping them, like by mapping `__meta_kube__meta_kubernetes_pod_label_foo` to `foo`. ```yaml - action: labelmap @@ -261,7 +261,7 @@ value or transformed to a final external label, such as `__job__`. Additional reading: - * [Julien Pivotto's slides from PromConf Munich, 2017](https://www.slideshare.net/roidelapluie/taking-advantage-of-prometheus-relabeling-109483749) + - [Julien Pivotto's slides from PromConf Munich, 2017](https://www.slideshare.net/roidelapluie/taking-advantage-of-prometheus-relabeling-109483749) ## HTTP client options diff --git a/docs/sources/clients/promtail/stages/_index.md b/docs/sources/clients/promtail/stages/_index.md index fa59e75b425e..1ee91611b13b 100644 --- a/docs/sources/clients/promtail/stages/_index.md +++ b/docs/sources/clients/promtail/stages/_index.md @@ -8,26 +8,26 @@ This section is a collection of all stages Promtail supports in a Parsing stages: - * [docker](docker/): Extract data by parsing the log line using the standard Docker format. - * [cri](cri/): Extract data by parsing the log line using the standard CRI format. - * [regex](regex/): Extract data using a regular expression. - * [json](json/): Extract data by parsing the log line as JSON. - * [replace](replace/): Replace data using a regular expression. + - [docker](docker/): Extract data by parsing the log line using the standard Docker format. + - [cri](cri/): Extract data by parsing the log line using the standard CRI format. + - [regex](regex/): Extract data using a regular expression. + - [json](json/): Extract data by parsing the log line as JSON. + - [replace](replace/): Replace data using a regular expression. Transform stages: - * [template](template/): Use Go templates to modify extracted data. + - [template](template/): Use Go templates to modify extracted data. Action stages: - * [timestamp](timestamp/): Set the timestamp value for the log entry. - * [output](output/): Set the log line text. - * [labeldrop](labeldrop/): Drop label set for the log entry. - * [labels](labels/): Update the label set for the log entry. - * [metrics](metrics/): Calculate metrics based on extracted data. - * [tenant](tenant/): Set the tenant ID value to use for the log entry. + - [timestamp](timestamp/): Set the timestamp value for the log entry. + - [output](output/): Set the log line text. + - [labeldrop](labeldrop/): Drop label set for the log entry. + - [labels](labels/): Update the label set for the log entry. + - [metrics](metrics/): Calculate metrics based on extracted data. + - [tenant](tenant/): Set the tenant ID value to use for the log entry. Filtering stages: - * [match](match/): Conditionally run stages based on the label set. - * [drop](drop/): Conditionally drop log lines based on several options. + - [match](match/): Conditionally run stages based on the label set. + - [drop](drop/): Conditionally drop log lines based on several options. diff --git a/docs/sources/community/governance.md b/docs/sources/community/governance.md index d535482548ee..3ec3d8981d6a 100644 --- a/docs/sources/community/governance.md +++ b/docs/sources/community/governance.md @@ -5,15 +5,15 @@ title: Governance This document describes the rules and governance of the project. It is meant to be followed by all the developers of the project and the Loki community. Common terminology used in this governance document are listed below: -* **Team members**: Any members of the private [team mailing list][team]. +- **Team members**: Any members of the private [team mailing list][team]. -* **Maintainers**: Maintainers lead an individual project or parts thereof ([`MAINTAINERS.md`][maintainers]). +- **Maintainers**: Maintainers lead an individual project or parts thereof ([`MAINTAINERS.md`][maintainers]). -* **Projects**: A single repository in the Grafana GitHub organization and listed below is referred to as a project: - * loki - * puppet-promtail +- **Projects**: A single repository in the Grafana GitHub organization and listed below is referred to as a project: + - loki + - puppet-promtail -* **The Loki project**: The sum of all activities performed under this governance, concerning one or more repositories or the community. +- **The Loki project**: The sum of all activities performed under this governance, concerning one or more repositories or the community. ## Values @@ -137,21 +137,21 @@ If there are multiple alternatives, members may vote for one or more alternative The new member is -* added to the list of [team members](#team-members). Ideally by sending a PR of their own, at least approving said PR. -* announced on the [developers mailing list][devs] by an existing team member. Ideally, the new member replies in this thread, acknowledging team membership. -* added to the projects with commit rights. -* added to the [team mailing list][team]. +- added to the list of [team members](#team-members). Ideally by sending a PR of their own, at least approving said PR. +- announced on the [developers mailing list][devs] by an existing team member. Ideally, the new member replies in this thread, acknowledging team membership. +- added to the projects with commit rights. +- added to the [team mailing list][team]. ### Offboarding The ex-member is -* removed from the list of [team members](#team-members). Ideally by sending a PR of their own, at least approving said PR. In case of forced removal, no approval is needed. -* removed from the projects. Optionally, they can retain maintainership of one or more repositories if the [team](#team-members) agrees. -* removed from the team mailing list and demoted to a normal member of the other mailing lists. -* not allowed to call themselves an active team member any more, nor allowed to imply this to be the case. -* added to a list of previous members if they so choose. +- removed from the list of [team members](#team-members). Ideally by sending a PR of their own, at least approving said PR. In case of forced removal, no approval is needed. +- removed from the projects. Optionally, they can retain maintainership of one or more repositories if the [team](#team-members) agrees. +- removed from the team mailing list and demoted to a normal member of the other mailing lists. +- not allowed to call themselves an active team member any more, nor allowed to imply this to be the case. +- added to a list of previous members if they so choose. If needed, we reserve the right to publicly announce removal. diff --git a/docs/sources/configuration/_index.md b/docs/sources/configuration/_index.md index 796525ea38ce..ffa06fd74a5f 100644 --- a/docs/sources/configuration/_index.md +++ b/docs/sources/configuration/_index.md @@ -66,15 +66,15 @@ non-list parameters the value is set to the specified default. Generic placeholders are defined as follows: -* `` : a boolean that can take the values `true` or `false` -* `` : any integer matching the regular expression `[1-9]+[0-9]*` -* `` : a duration matching the regular expression `[0-9]+(ns|us|µs|ms|[smh])` -* `` : a string matching the regular expression `[a-zA-Z_][a-zA-Z0-9_]*` -* `` : a string of unicode characters -* `` : a valid path relative to current working directory or an absolute path. -* `` : a valid string consisting of a hostname or IP followed by an optional port number -* `` : a regular string -* `` : a regular string that is a secret, such as a password +- `` : a boolean that can take the values `true` or `false` +- `` : any integer matching the regular expression `[1-9]+[0-9]*` +- `` : a duration matching the regular expression `[0-9]+(ns|us|µs|ms|[smh])` +- `` : a string matching the regular expression `[a-zA-Z_][a-zA-Z0-9_]*` +- `` : a string of unicode characters +- `` : a valid path relative to current working directory or an absolute path. +- `` : a valid string consisting of a hostname or IP followed by an optional port number +- `` : a regular string +- `` : a regular string that is a secret, such as a password Supported contents and default values of `loki.yaml`: diff --git a/docs/sources/design-documents/labels.md b/docs/sources/design-documents/labels.md index 96b95084638e..f1854d5a4ccc 100644 --- a/docs/sources/design-documents/labels.md +++ b/docs/sources/design-documents/labels.md @@ -19,18 +19,18 @@ Loki as a grep replacement, log tailing or log scrolling tool is highly desirabl As defined for prometheus “Use labels to differentiate the characteristics of the thing that is being measured” There are common cases where someone would want to search for all logs which had a level of “Error” or for a certain HTTP path (possibly too high cardinality), or of a certain order or event type. Examples: -* Log levels. -* HTTP Status codes. -* Event type. +- Log levels. +- HTTP Status codes. +- Event type. ## Challenges -* Logs are often unstructured data, it can be very difficult to extract reliable data from some unstructured formats, often requiring the use of complicated regular expressions. -* Easy to abuse. Easy to create a Label with high cardinality, even possibly by accident with a rogue regular expression. -* Where do we extract metrics and labels at the client (Promtail or other?) or Loki? Extraction at the server (Loki) side has some pros/cons. Can we do both? At least with labels we could define a set of expected labels and if loki doesn’t receive them they could be extracted. - * Server side extraction would improve interoperability at the expense of increase server workload and cost. - * Are there discoverability questions/concerns with metrics exposed via loki vs the agent? Maybe this is better/easier to manage? - * Potentially more difficult to manage configuration with the server side having to match configs to incoming log streams +- Logs are often unstructured data, it can be very difficult to extract reliable data from some unstructured formats, often requiring the use of complicated regular expressions. +- Easy to abuse. Easy to create a Label with high cardinality, even possibly by accident with a rogue regular expression. +- Where do we extract metrics and labels at the client (Promtail or other?) or Loki? Extraction at the server (Loki) side has some pros/cons. Can we do both? At least with labels we could define a set of expected labels and if loki doesn’t receive them they could be extracted. + - Server side extraction would improve interoperability at the expense of increase server workload and cost. + - Are there discoverability questions/concerns with metrics exposed via loki vs the agent? Maybe this is better/easier to manage? + - Potentially more difficult to manage configuration with the server side having to match configs to incoming log streams ## Existing Solutions @@ -178,15 +178,15 @@ You can define a more complicated regular expression with multiple capture group Please also note the regex for `message` is incomplete and would do a terrible job of matching any standard log message which might contain spaces or non alpha characters. ### Concerns -* Debugging, especially if a pipeline stage is mutating the log entry. -* Clashing labels and how to handle this (two stages try to set the same label) -* Performance vs ease of writing/use, if every label is extracted one at a time and there are a lot of labels and a long line, it would force reading the line many times, however contrast this to a really long complicated regex which only has to read the line once but is difficult to write and/or change and maintain +- Debugging, especially if a pipeline stage is mutating the log entry. +- Clashing labels and how to handle this (two stages try to set the same label) +- Performance vs ease of writing/use, if every label is extracted one at a time and there are a lot of labels and a long line, it would force reading the line many times, however contrast this to a really long complicated regex which only has to read the line once but is difficult to write and/or change and maintain ### Further improvements There are some basic building blocks for our pipeline which will use the EntryMiddleware interface, the two most commonly used will likely be: -* Regex Parser -* JSON Parser +- Regex Parser +- JSON Parser However we don’t want to ask people to copy and paste basic configs over and over for very common use cases, so it would make sense to add some additional parsers which would really be supersets of the base parsers above. @@ -237,6 +237,6 @@ There are also some challenges with auto detection and edge cases, though most p ## Other Thoughts and Considerations -* We should have a standalone client in some fashion which allows for testing of log parsing at the command line, allowing users to validate regular expressions or configurations to see what information is extracted. -* Other input formats to the pipeline which are not reading from log files, such as containerd grpc api, or from stdin or unix pipes, etc. -* It would be nice if at some point we could support loading code into the pipeline stages for even more advanced/powerful parsing capabilities. +- We should have a standalone client in some fashion which allows for testing of log parsing at the command line, allowing users to validate regular expressions or configurations to see what information is extracted. +- Other input formats to the pipeline which are not reading from log files, such as containerd grpc api, or from stdin or unix pipes, etc. +- It would be nice if at some point we could support loading code into the pipeline stages for even more advanced/powerful parsing capabilities. diff --git a/docs/sources/operations/loki-canary.md b/docs/sources/operations/loki-canary.md index 607a1ab8711f..8f79b9c29fc6 100644 --- a/docs/sources/operations/loki-canary.md +++ b/docs/sources/operations/loki-canary.md @@ -31,13 +31,13 @@ in the log message is compared to the internal array. If the received log is: -* The next in the array to be received, it is removed from the array and the +- The next in the array to be received, it is removed from the array and the (current time - log timestamp) is recorded in the `response_latency` histogram. This is the expected behavior for well behaving logs. -* Not the next in the array to be received, it is removed from the array, the +- Not the next in the array to be received, it is removed from the array, the response time is recorded in the `response_latency` histogram, and the `out_of_order_entries` counter is incremented. -* Not in the array at all, it is checked against a separate list of received +- Not in the array at all, it is checked against a separate list of received logs to either increment the `duplicate_entries` counter or the `unexpected_entries` counter. diff --git a/docs/sources/operations/storage/_index.md b/docs/sources/operations/storage/_index.md index 2bceae4c541b..728d07ee94b7 100644 --- a/docs/sources/operations/storage/_index.md +++ b/docs/sources/operations/storage/_index.md @@ -25,20 +25,20 @@ For more information: The following are supported for the index: -* [Amazon DynamoDB](https://aws.amazon.com/dynamodb) -* [Google Bigtable](https://cloud.google.com/bigtable) -* [Apache Cassandra](https://cassandra.apache.org) -* [BoltDB](https://github.com/boltdb/bolt) (doesn't work when clustering Loki) -* [BoltDB Shipper](boltdb-shipper/) EXPERIMENTAL index store which stores boltdb index files in the object store +- [Amazon DynamoDB](https://aws.amazon.com/dynamodb) +- [Google Bigtable](https://cloud.google.com/bigtable) +- [Apache Cassandra](https://cassandra.apache.org) +- [BoltDB](https://github.com/boltdb/bolt) (doesn't work when clustering Loki) +- [BoltDB Shipper](boltdb-shipper/) EXPERIMENTAL index store which stores boltdb index files in the object store The following are supported for the chunks: -* [Amazon DynamoDB](https://aws.amazon.com/dynamodb) -* [Google Bigtable](https://cloud.google.com/bigtable) -* [Apache Cassandra](https://cassandra.apache.org) -* [Amazon S3](https://aws.amazon.com/s3) -* [Google Cloud Storage](https://cloud.google.com/storage/) -* [Filesystem](filesystem/) (please read more about the filesystem to understand the pros/cons before using with production data) +- [Amazon DynamoDB](https://aws.amazon.com/dynamodb) +- [Google Bigtable](https://cloud.google.com/bigtable) +- [Apache Cassandra](https://cassandra.apache.org) +- [Amazon S3](https://aws.amazon.com/s3) +- [Google Cloud Storage](https://cloud.google.com/storage/) +- [Filesystem](filesystem/) (please read more about the filesystem to understand the pros/cons before using with production data) ## Cloud Storage Permissions @@ -46,9 +46,9 @@ The following are supported for the chunks: When using S3 as object storage, the following permissions are needed: -* `s3:ListBucket` -* `s3:PutObject` -* `s3:GetObject` +- `s3:ListBucket` +- `s3:PutObject` +- `s3:GetObject` Resources: `arn:aws:s3:::`, `arn:aws:s3:::/*` @@ -56,24 +56,24 @@ Resources: `arn:aws:s3:::`, `arn:aws:s3:::/*` When using DynamoDB for the index, the following permissions are needed: -* `dynamodb:BatchGetItem` -* `dynamodb:BatchWriteItem` -* `dynamodb:DeleteItem` -* `dynamodb:DescribeTable` -* `dynamodb:GetItem` -* `dynamodb:ListTagsOfResource` -* `dynamodb:PutItem` -* `dynamodb:Query` -* `dynamodb:TagResource` -* `dynamodb:UntagResource` -* `dynamodb:UpdateItem` -* `dynamodb:UpdateTable` -* `dynamodb:CreateTable` -* `dynamodb:DeleteTable` (if `table_manager.retention_period` is more than 0s) +- `dynamodb:BatchGetItem` +- `dynamodb:BatchWriteItem` +- `dynamodb:DeleteItem` +- `dynamodb:DescribeTable` +- `dynamodb:GetItem` +- `dynamodb:ListTagsOfResource` +- `dynamodb:PutItem` +- `dynamodb:Query` +- `dynamodb:TagResource` +- `dynamodb:UntagResource` +- `dynamodb:UpdateItem` +- `dynamodb:UpdateTable` +- `dynamodb:CreateTable` +- `dynamodb:DeleteTable` (if `table_manager.retention_period` is more than 0s) Resources: `arn:aws:dynamodb:::table/*` -* `dynamodb:ListTables` +- `dynamodb:ListTables` Resources: `*` @@ -83,19 +83,19 @@ If you enable autoscaling from table manager, the following permissions are need ##### Application Autoscaling -* `application-autoscaling:DescribeScalableTargets` -* `application-autoscaling:DescribeScalingPolicies` -* `application-autoscaling:RegisterScalableTarget` -* `application-autoscaling:DeregisterScalableTarget` -* `application-autoscaling:PutScalingPolicy` -* `application-autoscaling:DeleteScalingPolicy` +- `application-autoscaling:DescribeScalableTargets` +- `application-autoscaling:DescribeScalingPolicies` +- `application-autoscaling:RegisterScalableTarget` +- `application-autoscaling:DeregisterScalableTarget` +- `application-autoscaling:PutScalingPolicy` +- `application-autoscaling:DeleteScalingPolicy` Resources: `*` ##### IAM -* `iam:GetRole` -* `iam:PassRole` +- `iam:GetRole` +- `iam:PassRole` Resources: `arn:aws:iam:::role/` diff --git a/docs/sources/operations/upgrade.md b/docs/sources/operations/upgrade.md index cf2761a2fd76..41585a0bc1f6 100644 --- a/docs/sources/operations/upgrade.md +++ b/docs/sources/operations/upgrade.md @@ -333,13 +333,13 @@ kvstore: store: ``` -* Running without using a shared ring (inmemory): No action required -* Running with a shared ring and upgrading from v1.3.0 -> v1.4.0: No action required -* Running with a shared ring and upgrading from any version less than v1.3.0 (e.g. v1.2.0) -> v1.4.0: **ACTION REQUIRED** +- Running without using a shared ring (inmemory): No action required +- Running with a shared ring and upgrading from v1.3.0 -> v1.4.0: No action required +- Running with a shared ring and upgrading from any version less than v1.3.0 (e.g. v1.2.0) -> v1.4.0: **ACTION REQUIRED** There are two options for upgrade if you are not on version 1.3.0 and are using a shared ring: -* Upgrade first to v1.3.0 **BEFORE** upgrading to v1.4.0 +- Upgrade first to v1.3.0 **BEFORE** upgrading to v1.4.0 OR diff --git a/docs/sources/overview/_index.md b/docs/sources/overview/_index.md index c73c1bffd685..bcdc416d83b4 100644 --- a/docs/sources/overview/_index.md +++ b/docs/sources/overview/_index.md @@ -139,11 +139,11 @@ The **chunk store** is Loki's long-term data store, designed to support interactive querying and sustained writing without the need for background maintenance tasks. It consists of: -* An index for the chunks. This index can be backed by +- An index for the chunks. This index can be backed by [DynamoDB from Amazon Web Services](https://aws.amazon.com/dynamodb), [Bigtable from Google Cloud Platform](https://cloud.google.com/bigtable), or [Apache Cassandra](https://cassandra.apache.org). -* A key-value (KV) store for the chunk data itself, which can be DynamoDB, +- A key-value (KV) store for the chunk data itself, which can be DynamoDB, Bigtable, Cassandra again, or an object store such as [Amazon * S3](https://aws.amazon.com/s3) @@ -156,16 +156,16 @@ The chunk store relies on a unified interface to the Cassandra) that can be used to back the chunk store index. This interface assumes that the index is a collection of entries keyed by: -* A **hash key**. This is required for *all* reads and writes. -* A **range key**. This is required for writes and can be omitted for reads, +- A **hash key**. This is required for *all* reads and writes. +- A **range key**. This is required for writes and can be omitted for reads, which can be queried by prefix or range. The interface works somewhat differently across the supported databases: -* DynamoDB supports range and hash keys natively. Index entries are thus +- DynamoDB supports range and hash keys natively. Index entries are thus modelled directly as DynamoDB entries, with the hash key as the distribution key and the range as the range key. -* For Bigtable and Cassandra, index entries are modelled as individual column +- For Bigtable and Cassandra, index entries are modelled as individual column values. The hash key becomes the row key and the range key becomes the column key.