Skip to content

Commit

Permalink
Update admonitions to add shortcode
Browse files Browse the repository at this point in the history
  • Loading branch information
JStickler committed Jan 9, 2024
1 parent d7f945c commit 7d0db55
Show file tree
Hide file tree
Showing 17 changed files with 134 additions and 64 deletions.
4 changes: 3 additions & 1 deletion docs/sources/community/contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,10 @@ $ git commit -m "docs: fix spelling error"
$ git push -u fork HEAD
```

Note that if you downloaded Loki using `go get`, the message `package github.com/grafana/loki: no Go files in /go/src/github.com/grafana/loki`
{{% admonition type="note" %}}
If you downloaded Loki using `go get`, the message `package github.com/grafana/loki: no Go files in /go/src/github.com/grafana/loki`
is normal and requires no actions to resolve.
{{% /admonition %}}

### Building

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,10 @@ The performance losses against the current approach includes:

Loki regularly combines multiple blocks into a chunk and "flushes" it to storage. In order to ensure that reads over flushed chunks remain as performant as possible, we will re-order a possibly-overlapping set of blocks into a set of blocks that maintain monotonically increasing order between them. From the perspective of the rest of Loki’s components (queriers/rulers fetching chunks from storage), nothing has changed.

**Note: In the case that data for a stream is ingested in order, this is effectively a no-op, making it well optimized for in-order writes (which is both the requirement and default in Loki currently). Thus, this should have little performance impact on ordered data while enabling Loki to ingest unordered data.**
{{% admonition type="note" %}}
**In the case that data for a stream is ingested in order, this is effectively a no-op, making it well optimized for in-order writes (which is both the requirement and default in Loki currently). Thus, this should have little performance impact on ordered data while enabling Loki to ingest unordered data.**
{{% /admonition %}}


#### Chunk Durations

Expand All @@ -150,7 +153,9 @@ The second is simple to implement and an effective way to ensure Loki can ingest

We also cut chunks according to the `sync_period`. The first timestamp ingested past this bound will trigger a cut. This process aids in increasing chunk determinism and therefore our deduplication ratio in object storage because chunks are [content addressed](https://en.wikipedia.org/wiki/Content-addressable_storage). With the removal of our ordering constraint, it's possible that in some cases the synchronization method will not be as effective, such as during concurrent writes to the same stream across this bound.

**Note: It's important to mention that this is possible today with the current ordering constraint, but we'll be increasing the likelihood by removing it**
{{% admonition type="note" %}}
**It's important to mention that this is possible today with the current ordering constraint, but we'll be increasing the likelihood by removing it.**
{{% /admonition %}}

```
Figure 5
Expand Down
26 changes: 14 additions & 12 deletions docs/sources/community/maintaining/release-loki-build-image.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,19 +16,21 @@ if any changes were made in the folder `./loki-build-image/`.

## Step 1

1. create a branch with the desired changes to the Dockerfile
2. update the version tag of the `loki-build-image` pipeline defined in `.drone/drone.jsonnet` (search for `pipeline('loki-build-image')`) to a new version number (try to follow semver)
3. run `DRONE_SERVER=https://drone.grafana.net/ DRONE_TOKEN=<token> make drone` and commit the changes to the same branch
1. the `<token>` is your personal drone token, which can be found by navigating to https://drone.grafana.net/account.
4. create a PR
5. once approved and merged to `main`, the image with the new version is built and published
- **Note:** keep an eye on https://drone.grafana.net/grafana/loki for the build after merging ([example](https://drone.grafana.net/grafana/loki/17760/1/2))
1. Create a branch with the desired changes to the Dockerfile.
2. Update the version tag of the `loki-build-image` pipeline defined in `.drone/drone.jsonnet` (search for `pipeline('loki-build-image')`) to a new version number (try to follow semver).
3. Run `DRONE_SERVER=https://drone.grafana.net/ DRONE_TOKEN=<token> make drone` and commit the changes to the same branch.
1. The `<token>` is your personal drone token, which can be found by navigating to https://drone.grafana.net/account.
4. Create a PR.
5. Once approved and merged to `main`, the image with the new version is built and published.
{{% admonition type="note" %}}
Keep an eye on https://drone.grafana.net/grafana/loki for the build after merging ([example](https://drone.grafana.net/grafana/loki/17760/1/2)).
{{% /admonition %}}

## Step 2

1. create a branch
2. update the `BUILD_IMAGE_VERSION` variable in the `Makefile`
3. run `loki-build-image/version-updater.sh <new-version>` to update all the references
4. run `DRONE_SERVER=https://drone.grafana.net/ DRONE_TOKEN=<token> make drone` to update the Drone config to use the new build image
5. create a new PR
1. Create a branch.
2. Update the `BUILD_IMAGE_VERSION` variable in the `Makefile`.
3. Run `loki-build-image/version-updater.sh <new-version>` to update all the references.
4. Run `DRONE_SERVER=https://drone.grafana.net/ DRONE_TOKEN=<token> make drone` to update the Drone config to use the new build image.
5. Create a new PR.

5 changes: 4 additions & 1 deletion docs/sources/get-started/deployment-modes.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,11 +75,14 @@ For release 2.9 the components are:
- Ruler
- Table Manager (deprecated)

TIP: You can see the complete list of targets for your version of Loki by running Loki with the flag `-list-targets`, for example:
{{% admonition type="tip" %}}
You can see the complete list of targets for your version of Loki by running Loki with the flag `-list-targets`, for example:

```bash
docker run docker.io/grafana/loki:2.9.2 -config.file=/etc/loki/local-config.yaml -list-targets
```
{{% /admonition %}}

![Microservices mode diagram](../microservices-mode.png "Microservices mode")

Running components as individual microservices provides more granularity, letting you scale each component as individual microservices, to better match your specific use case.
Expand Down
9 changes: 3 additions & 6 deletions docs/sources/get-started/labels/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,13 +28,10 @@ See [structured metadata]({{< relref "./structured-metadata" >}}) for more infor

Loki places the same restrictions on label naming as [Prometheus](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels):

> It may contain ASCII letters and digits, as well as underscores and colons. It must match the regex `[a-zA-Z_:][a-zA-Z0-9_:]*`.
>
> Note: The colons are reserved for user defined recording rules. They should not be used by exporters or direct instrumentation.
- It may contain ASCII letters and digits, as well as underscores and colons. It must match the regex `[a-zA-Z_:][a-zA-Z0-9_:]*`.
- The colons are reserved for user defined recording rules. They should not be used by exporters or direct instrumentation.
- Unsupported characters in the label should be converted to an underscore. For example, the label `app.kubernetes.io/name` should be written as `app_kubernetes_io_name`.

{{% admonition type="note" %}}
Unsupported characters in the label should be converted to an underscore. For example, the label `app.kubernetes.io/name` should be written as `app_kubernetes_io_name`
{{% /admonition %}}

## Loki labels demo

Expand Down
5 changes: 3 additions & 2 deletions docs/sources/operations/authentication.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,11 @@ A list of open-source reverse proxies you can use:
- [OAuth2 proxy](https://github.com/oauth2-proxy/oauth2-proxy)
- [HAProxy](https://www.haproxy.org/)

Note that when using Loki in multi-tenant mode, Loki requires the HTTP header
{{% admonition type="note" %}}
When using Loki in multi-tenant mode, Loki requires the HTTP header
`X-Scope-OrgID` to be set to a string identifying the tenant; the responsibility
of populating this value should be handled by the authenticating reverse proxy.
For more information, read the [multi-tenancy]({{< relref "./multi-tenancy" >}}) documentation.
For more information, read the [multi-tenancy]({{< relref "./multi-tenancy" >}}) documentation.{{% /admonition %}}

For information on authenticating Promtail, see the documentation for [how to
configure Promtail]({{< relref "../send-data/promtail/configuration" >}}).
10 changes: 8 additions & 2 deletions docs/sources/operations/recording-rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,12 @@ is that Prometheus will, for example, reject a remote-write request with 100 sam
When the `ruler` starts up, it will load the WALs for the tenants who have recording rules. These WAL files are stored
on disk and are loaded into memory.

Note: WALs are loaded one at a time upon start-up. This is a current limitation of the Loki ruler.
{{% admonition type="note" %}}
WALs are loaded one at a time upon start-up. This is a current limitation of the Loki ruler.
For this reason, it is adviseable that the number of rule groups serviced by a ruler be kept to a reasonable size, since
_no rule evaluation occurs while WAL replay is in progress (this includes alerting rules)_.
{{% /admonition %}}


### Truncation

Expand All @@ -52,8 +55,11 @@ excessively large due to truncation.
## Scaling

See Mimir's guide for [configuring Grafana Mimir hash rings](/docs/mimir/latest/configure/configure-hash-rings/) for scaling the ruler using a ring.
Note: the `ruler` shards by rule _group_, not by individual rules. This is an artifact of the fact that Prometheus

{{% admonition type="note" %}}
The `ruler` shards by rule _group_, not by individual rules. This is an artifact of the fact that Prometheus
recording rules need to run in order since one recording rule can reuse another - but this is not possible in Loki.
{{% /admonition %}}

## Deployment

Expand Down
15 changes: 11 additions & 4 deletions docs/sources/operations/storage/boltdb-shipper.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ weight: 200
# Single Store BoltDB (boltdb-shipper)

{{% admonition type="note" %}}
Note that single store BoltDB Shipper is a legacy storage option and is not recommended for new deployments. The [TSDB]({{< relref "./tsdb" >}}) index is the recommended index.
Single store BoltDB Shipper is a legacy storage option and is not recommended for new deployments. The [TSDB]({{< relref "./tsdb" >}}) index is the recommended index.
{{% /admonition %}}

BoltDB Shipper lets you run Grafana Loki without any dependency on NoSQL stores for storing index.
Expand Down Expand Up @@ -75,7 +75,10 @@ they both having shipped files for day `18371` and `18372` with prefix `loki_ind
└── ingester-1-1587254400.gz
...
```
**Note:** We also add a timestamp to names of the files to randomize the names to avoid overwriting files when running Ingesters with same name and not have a persistent storage. Timestamps not shown here for simplification.

{{% admonition type="note" %}}
Loki also adds a timestamp to names of the files to randomize the names to avoid overwriting files when running Ingesters with same name and not have a persistent storage. Timestamps not shown here for simplification.
{{% /admonition %}}

Let us talk about more in depth about how both Ingesters and Queriers work when running them with BoltDB Shipper.

Expand All @@ -86,7 +89,9 @@ and the BoltDB Shipper looks for new and updated files in that directory at 1 mi
When running Loki in microservices mode, there could be multiple ingesters serving write requests.
Each ingester generates BoltDB files locally.

**Note:** To avoid any loss of index when an ingester crashes, we recommend running ingesters as a statefulset (when using Kubernetes) with a persistent storage for storing index files.
{{% admonition type="note" %}}
To avoid any loss of index when an ingester crashes, we recommend running ingesters as a StatefulSet (when using Kubernetes) with a persistent storage for storing index files.
{{% /admonition %}}

When chunks are flushed, they are available for reads in the object store instantly. The index is not available instantly, since we upload every 15 minutes with the BoltDB shipper.
Ingesters expose a new RPC for letting queriers query the ingester's local index for chunks which were recently flushed, but its index might not be available yet with queriers.
Expand Down Expand Up @@ -132,7 +137,9 @@ While using `boltdb-shipper` avoid configuring WriteDedupe cache since it is use
Compactor is a BoltDB Shipper specific service that reduces the index size by deduping the index and merging all the files to a single file per table.
We recommend running a Compactor since a single Ingester creates 96 files per day which include a lot of duplicate index entries and querying multiple files per table adds up the overall query latency.

**Note:** There should be only 1 compactor instance running at a time that otherwise could create problems and may lead to data loss.
{{% admonition type="note" %}}
There should be only one compactor instance running at a time that otherwise could create problems and may lead to data loss.
{{% /admonition %}}

Example compactor configuration with GCS:

Expand Down
2 changes: 2 additions & 0 deletions docs/sources/operations/storage/table-manager/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,9 +145,11 @@ number_of_tables_to_keep = floor(retention_period / table_period) + 1

![retention](./table-manager-retention.png)

{{% admonition type="note" %}}
It's important to note that - due to the internal implementation - the table
`period` and `retention_period` **must** be multiples of `24h` in order to get
the expected behavior.
{{% /admonition %}}

For detailed information on configuring the retention, refer to the
[Loki Storage Retention]({{< relref "../retention" >}})
Expand Down
4 changes: 2 additions & 2 deletions docs/sources/operations/storage/wal.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,13 @@ The Write Ahead Log in Loki takes a few particular tradeoffs compared to other W

In the event the WAL is corrupted/partially deleted, Loki will not be able to recover all of its data. In this case, Loki will attempt to recover any data it can, but will not prevent Loki from starting.

Note: the Prometheus metric `loki_ingester_wal_corruptions_total` can be used to track and alert when this happens.
You can use the Prometheus metric `loki_ingester_wal_corruptions_total` to track and alert when this happens.

1) No space left on disk

In the event the underlying WAL disk is full, Loki will not fail incoming writes, but neither will it log them to the WAL. In this case, the persistence guarantees across process restarts will not hold.

Note: the Prometheus metric `loki_ingester_wal_disk_full_failures_total` can be used to track and alert when this happens.
You can use the Prometheus metric `loki_ingester_wal_disk_full_failures_total` to track and alert when this happens.


### Backpressure
Expand Down
12 changes: 8 additions & 4 deletions docs/sources/query/logcli.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,9 +70,11 @@ without needing a username and password:
export LOKI_ADDR=http://localhost:3100
```

> Note: If you are running Loki behind a proxy server and you have
> authentication configured, you will also have to pass in LOKI_USERNAME
> and LOKI_PASSWORD, LOKI_BEARER_TOKEN or LOKI_BEARER_TOKEN_FILE accordingly.
{{% admonition type="note" %}}
If you are running Loki behind a proxy server and you have
authentication configured, you will also have to pass in LOKI_USERNAME
and LOKI_PASSWORD, LOKI_BEARER_TOKEN or LOKI_BEARER_TOKEN_FILE accordingly.
{{% /admonition %}}

```bash
$ logcli labels job
Expand Down Expand Up @@ -512,7 +514,9 @@ You can consume log lines from your `stdin` instead of Loki servers.

Say you have log files in your local, and just want to do run some LogQL queries for that, `--stdin` flag can help.

**NOTE: Currently it doesn't support any type of metric queries**
{{% admonition type="note" %}}
Currently it doesn't support any type of metric queries.
{{% /admonition %}}

You may have to use `stdin` flag for several reasons
1. Quick way to check and validate a LogQL expressions.
Expand Down
12 changes: 9 additions & 3 deletions docs/sources/send-data/lambda-promtail/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,9 @@ Ephemeral jobs can quite easily run afoul of cardinality best practices. During

For those using Cloudwatch and wishing to test out Loki in a low-risk way, this workflow allows piping Cloudwatch logs to Loki regardless of the event source (EC2, Kubernetes, Lambda, ECS, etc) without setting up a set of Promtail daemons across their infrastructure. However, running Promtail as a daemon on your infrastructure is the best-practice deployment strategy in the long term for flexibility, reliability, performance, and cost.

Note: Propagating logs from Cloudwatch to Loki means you'll still need to _pay_ for Cloudwatch.
{{% admonition type="note" %}}
Propagating logs from Cloudwatch to Loki means you'll still need to _pay_ for Cloudwatch.
{{% /admonition %}}

### VPC Flow logs

Expand Down Expand Up @@ -163,7 +165,9 @@ Incoming logs can have seven special labels assigned to them which can be used i

### Promtail labels

Note: This section is relevant if running Promtail between lambda-promtail and the end Loki deployment and was used to circumvent `out of order` problems prior to the v2.4 Loki release which removed the ordering constraint.
{{% admonition type="note" %}}
This section is relevant if running Promtail between lambda-promtail and the end Loki deployment and was used to circumvent `out of order` problems prior to the v2.4 Loki release which removed the ordering constraint.
{{% /admonition %}}

As stated earlier, this workflow moves the worst case stream cardinality from `number_of_log_streams` -> `number_of_log_groups` * `number_of_promtails`. For this reason, each Promtail must have a unique label attached to logs it processes (ideally via something like `--client.external-labels=promtail=${HOSTNAME}`) and it's advised to run a small number of Promtails behind a load balancer according to your throughput and redundancy needs.

Expand Down Expand Up @@ -191,7 +195,9 @@ The provided Terraform and CloudFormation files are meant to cover the default u

## Example Promtail Config

Note: this should be run in conjunction with a Promtail-specific label attached, ideally via a flag argument like `--client.external-labels=promtail=${HOSTNAME}`. It will receive writes via the push-api on ports `3500` (http) and `3600` (grpc).
{{% admonition type="note" %}}
This should be run in conjunction with a Promtail-specific label attached, ideally via a flag argument like `--client.external-labels=promtail=${HOSTNAME}`. It will receive writes via the push-api on ports `3500` (http) and `3600` (grpc).
{{% /admonition %}}

```yaml
server:
Expand Down
4 changes: 3 additions & 1 deletion docs/sources/send-data/promtail/stages/drop.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,9 @@ Would drop this log line:

#### Drop old log lines

**NOTE** For `older_than` to work, you must be using the [timestamp]({{< relref "./timestamp" >}}) stage to set the timestamp from the ingested log line _before_ applying the `drop` stage.
{{% admonition type="note" %}}
For `older_than` to work, you must be using the [timestamp]({{< relref "./timestamp" >}}) stage to set the timestamp from the ingested log line _before_ applying the `drop` stage.
{{% /admonition %}}

Given the pipeline:

Expand Down
4 changes: 3 additions & 1 deletion docs/sources/send-data/promtail/stages/json.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,5 +134,7 @@ The following key-value pairs would be created in the set of extracted data:
- `stream`: `stderr`
- `timestamp`: `2019-04-30T02:12:41.8443515`

Note that referring to `grpc.stream` without the combination of double quotes
{{% admonition type="note" %}}
Referring to `grpc.stream` without the combination of double quotes
wrapped in single quotes will not work properly.
{{% /admonition %}}
Loading

0 comments on commit 7d0db55

Please sign in to comment.