Releases: grafana/loki
v2.7.2
This is release v2.7.2 of Loki, a patch release of the v2.7 minor release.
Notable bug fixes:
- Fixed bug in validation of
patternandregexpparsers where missing or empty parameters for these parsers caused panics. - Fixed bugs in processing delete requests with line filters:
- Whole chunks covered by delete requests with a line filter were incorrectly skipped.
- Chunks partially covered by delete requests with a line filter were not properly processed.
The part that was not covered by the delete request was rewritten with a line filter, while the part that was covered by the delete request was completely deleted.
- Fixed bug in multi-tenant querying that caused HTTP 400 responses when multiple tenants where used in
X-Scope-OrgIDheader like sotenant-a|tenant-b. - Upgraded Go build version and Docker container base images to 1.19.5 to mitigate GO-2022-1144 vulnerability.
Installation:
The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.
Docker container:
$ docker pull "grafana/loki:2.7.2"
$ docker pull "grafana/promtail:2.7.2"Binary
We provide pre-compiled binary executables for the most common operating systems and architectures.
Choose from the assets below for the application and architecture matching your system.
Example for Loki on the linux operating system and amd64 architecture:
$ curl -O -L "https://github.com/grafana/loki/releases/download/v2.7.2/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"v2.7.1
trevorwhitney
Trevor Whitney
This is release v2.7.1 of Loki.
Notable changes:
- Add single compactor http client for delete and gennumber clients. This fixes a bug caused by the accidental introduction of different HTTP clients for compactor and gennumber operations that resulted in 404s when only the gennumber middlewares were enabled: ■ Unordered list style [Expected: dash; Actua
console Grot (@grafanabot), 2022-12-09 - [release-2.7.x] Prepare changelog and release notes for 2.7.1 (#7897)
level=error ts=2022-10-18T14:13:41.598649348Z caller=delete_requests_client.go:211 msg="error getting delete requests from the store" err="unexpected status code: 404"
ts=2022-10-18T14:13:41.598697295Z caller=spanlogger.go:80 user=application level=error msg="failed loading deletes for user" err="unexpected status code: 404"
- Change default deletion mode to
filter-onlyon account of the bug (see below). - Delete can now be performed over GRPC.
Installation:
The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.
Docker container:
$ docker pull "grafana/loki:2.7.1"
$ docker pull "grafana/promtail:2.7.1"Binary
We provide pre-compiled binary executables for the most common operating systems and architectures.
Choose from the assets below for the application and architecture matching your system.
Example for Loki on the linux operating system and amd64 architecture:
$ curl -O -L "https://github.com/grafana/loki/releases/download/v2.7.1/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"v2.7.0
trevorwhitney
Trevor Whitney
🦅 Loki 2.7.0 🦅
This is release v2.7.0 of Loki.
Notable changes:
- New Internal Server and TLS TLS can now be configured everywhere and there is an internal server to allow ring, memberlist, and readiness handlers to continue to work without TLS.
- Better Support for Azure Blob Storage thanks to the ability to use Azure's Service Principal Credentials.
- Logs can now be pushed from the Loki canary so you don't have to rely on a scraping service to use the canary.
- Additional
label_formatfields__timestamp__and__line__. - New embedded cache which is an in-process cache system that runs loki without the need for an external cache (like memcached, redis, etc).
- New HTTP endpoint for Ingester shutdown that will also delete the ring token.
- Faster label queries thanks to new parallization.
- Introducing Stream Sharding an experimental new feature to help deal with very large streams.
- Promtail
- support for max stream limit
- config reload endpoint / signal
- compressed file support
lambda-promtailnow supports Kinesis data stream events- matches for the journal reader
- basic tracing support
Upgrade Considerations
As always, please read the upgrade guide before upgrading Loki.
Installation:
The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.
Docker container:
$ docker pull "grafana/loki:2.7.0"
$ docker pull "grafana/promtail:2.7.0"Binary
We provide pre-compiled binary executables for the most common operating systems and architectures.
Choose from the assets below for the application and architecture matching your system.
Example for Loki on the linux operating system and amd64 architecture:
$ curl -O -L "https://github.com/grafana/loki/releases/download/v2.7.0/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"v2.6.1
vlad-diachenko
Vladyslav Diachenko
Loki 2.6.1 is a patch fix release on 2.6.0
Notable changes:
- PR 6658 JordanRushing: Updated the versions of dskit and memberlist to allow configuring cluster labels for memberlist. Cluster labels prevent mixing the members between two consistent hash rings of separate applications that are run on the same Kubernetes cluster.
- PR 6681 MasslessParticle Fixed an HTTP connection leak between the querier and the compactor when the log entry deletion feature is enabled.
- PR 6583 MasslessParticle Fixed noisy error messages when the log entry deletion feature is disabled for a tenant
Installation:
The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.
Docker container:
$ docker pull "grafana/loki:2.6.1"
$ docker pull "grafana/promtail:2.6.1"Binary
We provide pre-compiled binary executables for the most common operating systems and architectures.
Choose from the assets below for the application and architecture matching your system.
Example for Loki on the linux operating system and amd64 architecture:
$ curl -O -L "https://github.com/grafana/loki/releases/download/v2.6.1/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"v2.6.0
💪 Loki 2.6.0 💪
🦵:leg:
Features and enhancements
- Query multiple tenants at once. We've introduced cross-tenant query federation, which allows you to issue one query to multiple tenants and get a single, consolidated result. This is great for scenarios where you need a global view of logs within your multi-tenant cluster. For more information on how to enable this feature, see Multi-Tenancy.
- Filter out and delete certain log lines from query results. This is particularly useful in cases where users may accidentally write sensitive information to Loki that they do not want exposed. Users craft a LogQL query that selects the specific lines they're interested in, and then can choose to either filter out those lines from query results, or permanently delete them from Loki's storage. For more information, see Logs Deletion.
- Improved query performance on instant queries. Loki now splits instant queries with a large time range (for example,
sum(rate({app="foo"}[6h]))) into several smaller sub-queries and executes them in parallel. Users don't need to take any action to enjoy this performance improvement; however, they can adjust the number of sub-queries generated by modifying thesplit_queries_by_intervalconfiguration parameter, which currently defaults to30m. - Support Baidu AI Cloud as a storage backend. Loki users can now use Baidu Object Storage (BOS) as their storage backend. See bos_storage_config for details.
For a full list of all changes please look at the CHANGELOG.
Upgrade Considerations
As always, please read the upgrade guide before upgrading Loki.
Bug fixes
2.6.0 bug fixes
V2.6.0 fixes numerous bugs.
A summary of some of the more important fixes:
- PR 6152 Fixed a scenario where live tailing of logs could cause unbounded ingester memory growth.
- PR 5685 Fixed a bug in Loki's push request parser that allowed users to send arbitrary non-string data as a log line. We now test that the pushed values are valid strings and return an error if values are not valid strings.
- PR 5799 Fixed incorrect deduplication logic for cases where multiple log entries with the same timestamp exist.
- PR 5888 Fixed a bug in the common configuration where the
instance_interface_namessetting was getting overwritten by the default ring configuration.
Installation:
The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.
Docker container:
$ docker pull "grafana/loki:2.6.0"
$ docker pull "grafana/promtail:2.6.0"Binary
We provide pre-compiled binary executables for the most common operating systems and architectures.
Choose from the assets below for the application and architecture matching your system.
Example for Loki on the linux operating system and amd64 architecture:
$ curl -O -L "https://github.com/grafana/loki/releases/download/v2.6.0/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"v2.5.0
🦖 🧡 Loki 2.5.0 🧡 🦖
It has been nearly 6 months since Loki 2.4 was released, and we’ve been busy making Loki better than ever. Here's a summary of new enhancements and important fixes.
Features and enhancements
- Thanks to @bboreham for taking a deep dive on the Go
regexplibrary, and for creating a fork which greatly improves the performance of common regex uses cases in Loki. - Binary operations are now significantly faster, taking full advantage of Loki's parallelism.
- A new schema is available, which uses more path prefixes to avoid rate limits on S3.
- That same schema change was also added to the filesystem store, which avoids using one directory to store every chunk.
- A new capability for hedging requests to storage improves performance on highly parallelized queries. Refer to the hedging configuration block for more information.
- Promtail has several new ways to ingest logs:
- Promtail permits client-side global rate limiting and pipeline-configurable rate limiting.
For a full list of all changes please look at the CHANGELOG.
Upgrade Considerations
As always, please read the upgrade guide before upgrading Loki.
Changes to the config split_queries_by_interval
The most likely impact many people will see is Loki failing to start because of a change in the YAML configuration for split_queries_by_interval. It was previously possible to define this value in two places.
query_range:
split_queries_by_interval: 10mand/or
limits_config:
split_queries_by_interval: 10mIn 2.5.0 it can only be defined in the limits_config section.
Loki will fail to start if you do not remove the split_queries_by_interval configuration parameter from the query_range section.
Additionally, it has a new default value of 30m rather than 0.
The CLI flag is not changed and remains querier.split-queries-by-interval.
More parallelism by default
We continue to move defaults in Loki to leverage parallelism in all configurations, including the single binary.
In v2.5, all queries will be split and sharded by default. This will likely result in increased memory and CPU usage for Loki processes during queries, if you didn’t previously have these values enabled.
Usage reporting
Loki 2.5 includes code we added to report anonymous usage statistics back to Grafana Labs, an issue was created to outline the intent, and what went into the final implementation can be seen here in the source.
Usage reporting helps provide anonymous information on how people use Loki and what the Loki team should focus on for features and documentation. No private information is collected, and all reports are completely anonymous.
If possible, we ask you to leave the usage reporting feature enabled and help us understand more about Loki! We are also working to figure out how we can share this info with the community so everyone can watch Loki grow.
If you would rather not participate in usage stats reporting, the feature can be disabled in config
analytics:
reporting_enabled: false
Bug fixes
2.5.0 bug fixes
V2.5.0 fixes numerous bugs. The CHANGELOG has the complete list.
A summary of some of the more important fixes:
- Query correctness:
- PR 5474 cyriltovena: Disable sharding of count/avg when labels are mutated
- PR 5444 cyriltovena: Do not insert missing point when sharding
- PR 5423 cyriltovena: Correctly sets hash value for headblock iterator
- PR 5289 cyriltovena: Fixes log deduplication when mutating Labels using LogQL
- PR 5006 sandeepsukhani: Fix splitting of queries when step is larger than split interval
- PR 5413 owen-d: A deadlock in the Azure blob client
- PR 5334 kavirajk: A fix in live tailing that could lead to exploding memory usage
- PR 5144 owen-d: A fix for the ruler when trying to use Basic Auth for remote write
- Query cancellation:
- PR 4741 sandeepsukhani: An issue where retention didn't cleanup the index completely
Installation:
The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.
Docker container:
$ docker pull "grafana/loki:2.5.0"
$ docker pull "grafana/promtail:2.5.0"Binary
We provide pre-compiled binary executables for the most common operating systems and architectures.
Choose from the assets below for the application and architecture matching your system.
Example for Loki on the linux operating system and amd64 architecture:
$ curl -O -L "https://github.com/grafana/loki/releases/download/v2.5.0/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"Contributors
Assets 34
v2.4.2
Loki 2.4.2
Loki 2.4.2 is a patch fix release on 2.4.x
Defaults changes
2.4.2 makes the following changes to Loki defaults to improve usability, see PR 5077:
| config | new default | old default |
|---|---|---|
| parallelise_shardable_queries | true | false |
| split_queries_by_interval | 30m | 0s |
| query_ingesters_within | 3h | 0s |
| max_chunk_age | 2h | 1h |
| max_concurrent | 10 | 20 |
Bug fixes
2.4.2 fixes these bugs:
- PR 4968 trevorwhitney: Fixes a bug in which querying ingesters wrongly returns a ruler,
causing the internal server errorcode = Unimplemented. - PR 4875 trevorwhitney: Honor the replication factor specified in the common configuration block when
memberlistis the consistent hash ring store. - PR 4792 AndreZiviani: Corrects the default values of configuration options in the documentation for:
scheduler_dns_lookup_periodmin_ready_durationfinal_sleepmax_transfer_retrieschunk_retain_periodchunk_target_sizebatch_sizetimeout(for Redis requests)
Installation:
The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.
Docker container:
$ docker pull "grafana/loki:2.4.2"
$ docker pull "grafana/promtail:2.4.2"Binary
We provide pre-compiled binary executables for the most common operating systems and architectures.
Choose from the assets below for the application and architecture matching your system.
Example for Loki on the linux operating system and amd64 architecture:
$ curl -O -L "https://github.com/grafana/loki/releases/download/v2.4.2/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"v2.4.1
Loki 2.4.1
This is a patch fix release which fixes two important bugs found in the 2.4.0 release:
- PR 4687 owen-d: Fixes a compactor panic when not using a tenant overrides file.
- PR 4681 slim-bean: Fixes an incorrect initialization of the
readtarget resulting in query gaps after a chunk is flushed and before the querier would download new index tables.
Installation:
The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.
Docker container:
$ docker pull "grafana/loki:2.4.1"
$ docker pull "grafana/promtail:2.4.1"Binary
We provide pre-compiled binary executables for the most common operating systems and architectures.
Choose from the assets below for the application and architecture matching your system.
Example for Loki on the linux operating system and amd64 architecture:
$ curl -O -L "https://github.com/grafana/loki/releases/download/v2.4.1/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"v2.4.0
🎊 🌮 Loki 2.4.0 🌮 🎊
With Loki 2.4 we’ve focused on 2 main things:
- Making it easier than ever to get your logs into Loki.
- Making it easier to run and operate, anyone not using Kubernetes or allergic to microservices should pay close attention, this release has a lot for you.
Features and enhancements
- Loki no longer requires logs to be sent in perfect chronological order. Support for out of order logs is one of the most highly requested features for Loki and in 2.4 the strict ordering constraint has been removed.
- Scaling Loki is now much easier with a hybrid mode between a single binary and microservices we are calling the Simple Scalable Deployment. It’s now possible to scale Loki with new
readandwritetargets. Where previously you would have needed Kubernetes and the microservices approach to start tapping into Loki’s potential, it’s now possible to do this in a much simpler way. - We introduced a common config section and updated and updated a lot of defaults. This results in a 70% smaller Loki config and comes out of the box with more appropriate defaults and limits. Check out the example local config as the new reference for running Loki.
- Recording rules are no longer marked experimental and have a much more resilient implementation which leverages the existing write ahead log code in Prometheus.
- Do you use Kafka for your logs? You should check out the new Promtail Kafka Consumer which can easily get your logs out of Kafka and into Loki.
- LogQL has received some nice enhancements thanks to the amazing Loki community in the form of group_left and group_right as well as the
label_formatandline_formatfunctions now support working with dates and times - Another great community contribution to Promtail allows it to accept ndjson and plaintext log files over HTTP.
All in all about 260 PR’s went into Loki 2.4 and we thank everyone for helping us make the best Loki yet.
For a full list of all changes please look at the CHANGELOG
Upgrade Considerations
Please read the upgrade guide before updating Loki.
We made a lot of changes to Loki’s configuration as part of this release, we have tried our best to make sure changes are compatible with existing configurations however some changes to default limits may impact users who didn't have values explicitly set for these limits in their config files.
Here is a list of limits changed:
| config | new default | old default |
|---|---|---|
| ingestion_rate_strategy | "global" | "local" |
| max_global_streams_per_user | 5000 | 0 (no limit) |
| max_query_length | "721h" | "0h" (no limit) |
| max_query_parallelism | 32 | 14 |
| max_streams_per_user | 0 (no limit) | 10000 |
| reject_old_samples | true | false |
| reject_old_samples_max_age | "168h" | "336h" |
| per_stream_rate_limit | 3MB | - |
| per_stream_rate_limit_burst | 15MB | - |
Also we have enabled the in-memory FIFO cache by default when no existing external cache is configured. This helps improve Loki's performance however it will also allow Loki to consume more memory.
Bug fixes
Lists of bug fixes for 2.4.x.
2.4.0 bug fixes
Here is a list of some of the more important fixes in 2.4.0:
- 4598 kavirajk: Fix
ipmatcher lexer to differentiate filter from identifier - 4563 cyriltovena: Fixes the Series function to handle properly sharding.
- 4518 slim-bean: Loki: Fix bug where items are returned to a sync.Pool incorrectly
- 4411 slim-bean: Loki: Bug: frontend waiting on results which would never come
- 4238 liguozhong: [fix] distributor: fix goroutine leak
Security fixes
List of security fixes for 2.4.x
2.4.0 security fixes
2.4.0 contains one fix which is security related
4627 Loki: Explicitly define allowed HTTP methods on HTTP endpoints
A community user noticed that all Loki endpoints would respond to HTTP OPTIONS requests, and the proxy they placed in front of Loki responsible for HTTP authentication would pass OPTIONS requests unauthenticated to Loki allowing users to make unauthenticated/unauthorized queries to Loki. This PR restricts the type of HTTP methods allowed on each endpoint and disallows OPTIONS requests.
Installation:
The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.
Docker container:
$ docker pull "grafana/loki:2.4.0"
$ docker pull "grafana/promtail:2.4.0"Binary
We provide pre-compiled binary executables for the most common operating systems and architectures.
Choose from the assets below for the application and architecture matching your system.
Example for Loki on the linux operating system and amd64 architecture:
$ curl -O -L "https://github.com/grafana/loki/releases/download/v2.4.0/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"v2.3.0
🥳🎉 Loki 2.3.0 🎉🥳
The Loki team is excited to announce the release of Loki 2.3!
It's been nearly 6 months since 2.2 was released and we have made good use of that time to bring forward several significant improvements and requested features.
2.3 is also the first version of Loki released under the AGPLv3 license. You can read more about our licensing here.
Some parts of the Loki repo will remain Apache-2.0 licensed (mainly clients and some tooling), for more details please read LICENSING.md.
Features and enhancements
- Loki now has the ability to apply custom retention based on stream selectors! This will allow much finer control over log retention all of which is now handled by Loki, no longer requiring the use of object store configs for retention.
- Coming along hand in hand with storing logs for longer durations is the ability to delete log streams. The initial implementation lets you submit delete request jobs which will be processed after 24 hours.
- A very exciting new LogQL parser has been introduced: the pattern parser. Much simpler and faster than regexp for log lines that have a little bit of structure to them such as the Common Log Format. This is now Loki's fastest parser so try it out on any of your log lines!
- Extending on the work of Alerting Rules, Loki now accepts recording rules. This lets you turn your logs into metrics and push them to Prometheus or any Prometheus compatible remote_write endpoint.
- LogQL can understand IP addresses! This enables filtering on IP addresses and subnet ranges.
For those of you running Loki as microservices, the following features will improve performance operations significantly for many operations.
- We created an index gateway which takes on the task of downloading the boltdb-shipper index files allowing you to run your queriers without any local disk requirements, this is really helpful in Kubernetes environments where you can return your queriers from Statefulsets back to Deployments and save a lot of PVC costs and operational headaches.
- Ingester queriers are now shardable, this is a significant performance boost for high volume log streams when querying recent data.
- Instant queries can now be split and sharded making them just as fast as range queries.
A very common feature requested has also been included in 2.3:
- 3919 github-vincent-miszczak: Add ingester.autoforget-unhealthy-timeout opt-in feature
Without revisiting the decisions and discussions around the somewhat controversial behavior of unhealthy ingesters, you can now decided how you would like them to be handled: manually or automatically.
Lastly several useful additions to the LogQL query language have been included:
- More text/template functions are included for
label_formatandline_formatwith PR 3515, please check out the documentation for template functions. - Also support for math functions withing
label_formatandline_formatwas included with 3434. - Two additional metric functions with some interesting use cases
first_over_timeandlast_over_timewere added in PR 3050. These can be useful for some down sampling approaches where instead of taking an average, max, or min of samples over a range in a metrics query, you can select the first or last log line to use from that range.
For the full list of changes (almost 300!!) check out the CHANGELOG
Upgrade considerations
The path from 2.2.1 to 2.3.0 should be smooth, as always, please check out the Upgrade Guide for important upgrade guidance.
One change we consider noteworthy however is:
- 3216 sandeepsukhani: check for stream selectors to have at least one equality matcher.
This change now rejects any query which does not contain at least one equality matcher, an example may better illustrate:
{namespace=~".*"}
This query will now be rejected, however there are several ways to modify it for it to succeed:
Add at least one equals label matcher:
{cluster="us-east-1",namespace=~".*"}
Use .+ instead of .*
{namespace=~".+"}
This difference may seem subtle but if we break it down . matches any character, * matches zero or more of the preceding character and + matches one or more of the preceding character. The .* case will match empty values where .+ will not, this is the important difference. {namespace=""} is an invalid request (unless you add another equals label matcher like the example above).
The reasoning for this change has to do with how index lookups work in Loki, if you don't have at least one equality matcher Loki has to perform a complete index table scan which is an expensive and slow operation.
Security fixes
List of security fixes for 2.3.x.
2.3.0 security fixes
2.3.0 contains an important security fix:
- 4020 simonswine: Restrict path segments in TenantIDs (CVE-2021-36156 CVE-2021-36157).
Note Exploitation of this vulnerability requires the ability for an attacker to craft and send directly to Loki an X-Scope-OrgID header, end users should not have the ability to create and send this header directly to Loki as it controls access to tenants and is important to control setting of this header for proper tenant isolation and security. We always recommend having a proxy or gateway be responsible for setting the X-Scope-OrgID.
Bug fixes
Lists of bug fixes for 2.3.x.
2.3.0 bug fixes
- An important fix for leaking resources was patched with 3733, when queries were canceled a goroutine was left running which would hold memory resources creating a memory leak.
- 3686 fixes a panic with the frontend when use with downstream URL. Note we recommend using the GRPC Pull Model, better performance and fair scheduling between tenants can be obtained with the GPRC Pull Model.
Installation:
The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.
Docker container:
$ docker pull "grafana/loki:2.3.0"
$ docker pull "grafana/promtail:2.3.0"Binary
We provide pre-compiled binary executables for the most common operating systems and architectures.
Choose from the assets below for the application and architecture matching your system.
Example for Loki on the linux operating system and amd64 architecture:
$ curl -O -L "https://github.com/grafana/loki/releases/download/v2.3.0/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"