From 404d54445de6a37b31df459ee18c88bf39ef8460 Mon Sep 17 00:00:00 2001 From: Edward Welch Date: Thu, 11 Mar 2021 07:58:30 -0500 Subject: [PATCH 1/9] 2.2 release notes, changelog and upgrade notes --- CHANGELOG.md | 266 ++++++++++++++++++++++++++++ docs/sources/installation/docker.md | 18 +- docs/sources/upgrading/_index.md | 25 ++- 3 files changed, 296 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9c4ef9265e9a..27f4acdb38ec 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,269 @@ +## 2.2.0 (2021/03/10) + +With over 200 PR's 2.2 includes significant features, performance improvements, and bug fixes! + +The most upvoted issue for Loki was closed in this release! [Issue 74](https://github.com/grafana/loki/issues/74) requesting support for handling multi-line logs in Promtail was implemented in [PR 3024](https://github.com/grafana/loki/pull/3024). Thanks @jeschkies! + +Other exciting news for Promtail, [PR 3246](https://github.com/grafana/loki/pull/3246) by @cyriltovena introduces support for reading Windows Events! + +Switching to Loki, @owen-d has added a write ahead log to Loki! [PR 2981](https://github.com/grafana/loki/pull/2981) was the first of many as we have spent the last several months using and abusing our write ahead logs to flush out any bugs! + +A significant number of the PR's in this release have gone to improving the features introduced in Loki 2.0. @cyriltovena overhauled the JSON parser in [PR 3163](https://github.com/grafana/loki/pull/3163) (and a few other PR's), to provide both a faster and smarter parsing to only extract JSON content which is used in the query output. The newest Loki squad member @dannykopping fine tuned the JSON parser options in [PR 3280](https://github.com/grafana/loki/pull/3280) allowing you to specific individual JSON elements, including support now for accessing elements in an array. Many, many other additional improvements have been made, as well as several fixes to the new LogQL features added some months ago, this upgrade should have everyone seeing improvements in their queries. + +@cyriltovena also set his PPROF skills loose on the Loki write path which resulted in about 8x less memory usage on our distributors and a much more stable memory usage when ingesters are flushing a lot of chunks at the same time. + +There are many other noteworthy additions and fixes, too many to list, but we should call out one more feature all you Google Cloud Platform users might be excited about: in [PR 3083](https://github.com/grafana/loki/pull/3083) @kavirajk added support to Promtail for listening on Google Pub/Sub topics, letting you setup log sinks for your GCP logs to be ingested by Promtail and sent to Loki! + +Thanks to everyone for another exciting Loki release!! + +Please read the [Upgrade Guide](https://github.com/grafana/loki/blob/master/docs/sources/upgrading/_index.md#220) before upgrading for a smooth experience. + +TL;DR Loki 2.2 changes the internal chunk format which limits what versions you can downgrade to, and we fixed a mistake related `scrape_configs` which do not have a `pipeline_stages` defined. If you have any Promtail `scrape_configs` which do not specify `pipeline_stages` you should go read the upgrade notes! + +### All Changes + +#### Loki + +* [3460](https://github.com/grafana/loki/pull/3460) **slim-bean**: Loki: Per Tenant Runtime Configs +* [3459](https://github.com/grafana/loki/pull/3459) **cyriltovena**: Fixes split interval for metrics queries. +* [3432](https://github.com/grafana/loki/pull/3432) **slim-bean**: Loki: change ReadStringAsSlice to ReadString so it doesn't parse quotes inside the packed _entry +* [3429](https://github.com/grafana/loki/pull/3429) **cyriltovena**: Improve the parser hint tests. +* [3426](https://github.com/grafana/loki/pull/3426) **cyriltovena**: Only unpack entry if the key `_entry` exist. +* [3424](https://github.com/grafana/loki/pull/3424) **cyriltovena**: Add fgprof to Loki and Promtail. +* [3423](https://github.com/grafana/loki/pull/3423) **cyriltovena**: Add limit and line_returned in the query log. +* [3420](https://github.com/grafana/loki/pull/3420) **cyriltovena**: Introduce a unpack parser. +* [3417](https://github.com/grafana/loki/pull/3417) **cyriltovena**: Fixes a race in the scheduling limits. +* [3416](https://github.com/grafana/loki/pull/3416) **ukolovda**: Distributor: append several tests for HTTP parser. +* [3411](https://github.com/grafana/loki/pull/3411) **slim-bean**: Loki: fix alignment of atomic 64 bit to work with 32 bit OS +* [3409](https://github.com/grafana/loki/pull/3409) **gotjosh**: Instrumentation: Add histogram for request duration on gRPC client to Ingesters +* [3408](https://github.com/grafana/loki/pull/3408) **jgehrcke**: distributor: fix snappy-compressed protobuf POST request handling (#3407) +* [3388](https://github.com/grafana/loki/pull/3388) **owen-d**: prevents duplicate log lines from being replayed. closes #3378 +* [3383](https://github.com/grafana/loki/pull/3383) **cyriltovena**: Fixes head chunk iterator direction. +* [3380](https://github.com/grafana/loki/pull/3380) **slim-bean**: Loki: Fix parser hint for extracted labels which collide with stream labels +* [3372](https://github.com/grafana/loki/pull/3372) **cyriltovena**: Fixes a panic with whitespace key. +* [3350](https://github.com/grafana/loki/pull/3350) **cyriltovena**: Fixes ingester stats. +* [3348](https://github.com/grafana/loki/pull/3348) **cyriltovena**: Fixes 500 in the querier when returning multiple errors. +* [3347](https://github.com/grafana/loki/pull/3347) **cyriltovena**: Fixes a tight loop in the Engine with LogQL parser. +* [3344](https://github.com/grafana/loki/pull/3344) **cyriltovena**: Fixes some 500 returned by querier when storage cancellation happens. +* [3342](https://github.com/grafana/loki/pull/3342) **cyriltovena**: Bound parallelism frontend +* [3340](https://github.com/grafana/loki/pull/3340) **owen-d**: Adds some flushing instrumentation/logs +* [3339](https://github.com/grafana/loki/pull/3339) **owen-d**: adds Start() method to WAL interface to delay checkpointing until aft… +* [3338](https://github.com/grafana/loki/pull/3338) **sandeepsukhani**: dedupe index on all the queries for a table instead of query batches +* [3326](https://github.com/grafana/loki/pull/3326) **owen-d**: removes wal recover flag +* [3307](https://github.com/grafana/loki/pull/3307) **slim-bean**: Loki: fix validation error and metrics +* [3306](https://github.com/grafana/loki/pull/3306) **cyriltovena**: Add finalizer to zstd. +* [3300](https://github.com/grafana/loki/pull/3300) **sandeepsukhani**: increase db retain period in ingesters to cover index cache validity period as well +* [3299](https://github.com/grafana/loki/pull/3299) **owen-d**: Logql/absent label optimization +* [3295](https://github.com/grafana/loki/pull/3295) **jtlisi**: chore: update cortex to latest and fix refs +* [3291](https://github.com/grafana/loki/pull/3291) **ukolovda**: Distributor: Loki API can receive gzipped JSON +* [3280](https://github.com/grafana/loki/pull/3280) **dannykopping**: LogQL: Simple JSON expressions +* [3279](https://github.com/grafana/loki/pull/3279) **cyriltovena**: Fixes logfmt parser hints. +* [3278](https://github.com/grafana/loki/pull/3278) **owen-d**: Testware/ rate-unwrap-multi +* [3274](https://github.com/grafana/loki/pull/3274) **liguozhong**: [ingester_query] change var "clients" to "reps" +* [3267](https://github.com/grafana/loki/pull/3267) **jeschkies**: Update vendored Cortex to 0976147451ee +* [3263](https://github.com/grafana/loki/pull/3263) **cyriltovena**: Fix a bug with metric queries and label_format. +* [3261](https://github.com/grafana/loki/pull/3261) **sandeepsukhani**: fix broken json logs push path +* [3256](https://github.com/grafana/loki/pull/3256) **jtlisi**: update vendored cortex and add new replace overrides +* [3251](https://github.com/grafana/loki/pull/3251) **cyriltovena**: Ensure we have parentheses for bin ops. +* [3249](https://github.com/grafana/loki/pull/3249) **cyriltovena**: Fixes a bug where slice of Entries where not zeroed +* [3241](https://github.com/grafana/loki/pull/3241) **cyriltovena**: Allocate entries array with correct size while decoding WAL entries. +* [3237](https://github.com/grafana/loki/pull/3237) **cyriltovena**: Fixes unmarshalling of tailing responses. +* [3236](https://github.com/grafana/loki/pull/3236) **slim-bean**: Loki: Log a crude lag metric for how far behind a client is. +* [3234](https://github.com/grafana/loki/pull/3234) **cyriltovena**: Fixes previous commit not using the new sized body. +* [3233](https://github.com/grafana/loki/pull/3233) **cyriltovena**: Re-introduce https://github.com/grafana/loki/pull/3178. +* [3228](https://github.com/grafana/loki/pull/3228) **MichelHollands**: Add config endpoint +* [3218](https://github.com/grafana/loki/pull/3218) **owen-d**: WAL backpressure +* [3217](https://github.com/grafana/loki/pull/3217) **cyriltovena**: Rename checkpoint proto package to avoid conflict with cortex. +* [3215](https://github.com/grafana/loki/pull/3215) **cyriltovena**: Cortex update pre 1.7 +* [3211](https://github.com/grafana/loki/pull/3211) **cyriltovena**: Fixes tail api marshalling for v1. +* [3210](https://github.com/grafana/loki/pull/3210) **cyriltovena**: Reverts flush buffer pooling. +* [3201](https://github.com/grafana/loki/pull/3201) **sandeepsukhani**: limit query range in async store for ingesters when query-ingesters-within flag is set +* [3200](https://github.com/grafana/loki/pull/3200) **cyriltovena**: Improve ingester flush memory usage. +* [3195](https://github.com/grafana/loki/pull/3195) **owen-d**: Ignore flushed chunks during checkpointing +* [3194](https://github.com/grafana/loki/pull/3194) **cyriltovena**: Fixes unwrap expressions from last optimization. +* [3193](https://github.com/grafana/loki/pull/3193) **cyriltovena**: Improve checkpoint series iterator. +* [3188](https://github.com/grafana/loki/pull/3188) **cyriltovena**: Improves checkpointerWriter memory usage +* [3180](https://github.com/grafana/loki/pull/3180) **owen-d**: moves boltdb flags to config file +* [3178](https://github.com/grafana/loki/pull/3178) **cyriltovena**: Logs PushRequest data. +* [3163](https://github.com/grafana/loki/pull/3163) **cyriltovena**: Uses custom json-iter decoder for log entries. +* [3159](https://github.com/grafana/loki/pull/3159) **MichelHollands**: Make httpAuthMiddleware field public +* [3153](https://github.com/grafana/loki/pull/3153) **cyriltovena**: Improve wal entries encoding. +* [3152](https://github.com/grafana/loki/pull/3152) **AllenzhLi**: update github.com/gorilla/websocket to fixes a potential denial-of-service (DoS) vector +* [3146](https://github.com/grafana/loki/pull/3146) **owen-d**: More semantically correct flush shutdown +* [3143](https://github.com/grafana/loki/pull/3143) **cyriltovena**: Fixes absent_over_time to work with all log selector. +* [3141](https://github.com/grafana/loki/pull/3141) **owen-d**: Swaps mutex for atomic in ingester's OnceSwitch +* [3137](https://github.com/grafana/loki/pull/3137) **owen-d**: label_format no longer shardable and introduces the Shardable() metho… +* [3136](https://github.com/grafana/loki/pull/3136) **owen-d**: Don't fail writes due to full WAL disk +* [3134](https://github.com/grafana/loki/pull/3134) **cyriltovena**: Improve distributors validation and apply in-place filtering. +* [3132](https://github.com/grafana/loki/pull/3132) **owen-d**: Integrates label replace into sharding code +* [3131](https://github.com/grafana/loki/pull/3131) **MichelHollands**: Update cortex to 1 6 +* [3126](https://github.com/grafana/loki/pull/3126) **dannykopping**: Implementing line comments +* [3122](https://github.com/grafana/loki/pull/3122) **owen-d**: Self documenting pipeline process interface +* [3117](https://github.com/grafana/loki/pull/3117) **owen-d**: Wal/recover corruption +* [3114](https://github.com/grafana/loki/pull/3114) **owen-d**: Disables the stream limiter until wal has recovered +* [3092](https://github.com/grafana/loki/pull/3092) **liguozhong**: lru cache logql.ParseLabels +* [3090](https://github.com/grafana/loki/pull/3090) **cyriltovena**: Improve tailer matching by using the index. +* [3087](https://github.com/grafana/loki/pull/3087) **MichelHollands**: feature: make server publicly available +* [3080](https://github.com/grafana/loki/pull/3080) **cyriltovena**: Improve JSON parser and add labels parser hints. +* [3077](https://github.com/grafana/loki/pull/3077) **MichelHollands**: Make the moduleManager field public +* [3065](https://github.com/grafana/loki/pull/3065) **cyriltovena**: Optimizes SampleExpr to remove unnecessary line_format. +* [3064](https://github.com/grafana/loki/pull/3064) **cyriltovena**: Add zstd and flate compressions algorithms. +* [3053](https://github.com/grafana/loki/pull/3053) **cyriltovena**: Add absent_over_time +* [3048](https://github.com/grafana/loki/pull/3048) **cyriltovena**: Support rate for unwrapped expressions. +* [3047](https://github.com/grafana/loki/pull/3047) **cyriltovena**: Add function label_replace. +* [3030](https://github.com/grafana/loki/pull/3030) **cyriltovena**: Allows by/without to be empty and available for max/min_over_time +* [3025](https://github.com/grafana/loki/pull/3025) **cyriltovena**: Fixes a swallowed context err in the batch storage. +* [3013](https://github.com/grafana/loki/pull/3013) **owen-d**: headblock checkpointing up to v3 +* [3008](https://github.com/grafana/loki/pull/3008) **cyriltovena**: Fixes the ruler storage with the boltdb store. +* [3000](https://github.com/grafana/loki/pull/3000) **owen-d**: Introduces per stream chunks mutex +* [2981](https://github.com/grafana/loki/pull/2981) **owen-d**: Adds WAL support (experimental) +* [2960](https://github.com/grafana/loki/pull/2960) **sandeepsukhani**: fix table deletion in table client for boltdb-shipper + +#### Promtail + +* [3422](https://github.com/grafana/loki/pull/3422) **kavirajk**: Modify script to accept inclusion and exclustion filters as variables +* [3404](https://github.com/grafana/loki/pull/3404) **dannykopping**: Remove default docker pipeline stage +* [3401](https://github.com/grafana/loki/pull/3401) **slim-bean**: Promtail: Add pack stage +* [3381](https://github.com/grafana/loki/pull/3381) **adityacs**: fix nested captured groups indexing in replace stage +* [3332](https://github.com/grafana/loki/pull/3332) **cyriltovena**: Embed timezone data in Promtail. +* [3304](https://github.com/grafana/loki/pull/3304) **kavirajk**: Use project-id from the variables. Remove hardcoding +* [3303](https://github.com/grafana/loki/pull/3303) **cyriltovena**: Increase the windows bookmark buffer. +* [3302](https://github.com/grafana/loki/pull/3302) **cyriltovena**: Fixes races in multiline stage and promtail. +* [3298](https://github.com/grafana/loki/pull/3298) **gregorybrzeski**: Promtail: fix typo in config variable name - BookmarkPath +* [3285](https://github.com/grafana/loki/pull/3285) **kavirajk**: Make incoming labels from gcp into Loki internal labels. +* [3284](https://github.com/grafana/loki/pull/3284) **kavirajk**: Avoid putting all the GCP labels into loki labels +* [3246](https://github.com/grafana/loki/pull/3246) **cyriltovena**: Windows events +* [3224](https://github.com/grafana/loki/pull/3224) **veltmanj**: Fix(pkg/promtail) CVE-2020-11022 JQuery vulnerability +* [3207](https://github.com/grafana/loki/pull/3207) **cyriltovena**: Fixes panic when using multiple clients +* [3191](https://github.com/grafana/loki/pull/3191) **rfratto**: promtail: pass registerer to gcplog +* [3175](https://github.com/grafana/loki/pull/3175) **rfratto**: Promtail: pass a prometheus registerer to promtail components +* [3083](https://github.com/grafana/loki/pull/3083) **kavirajk**: Gcplog targetmanager +* [3024](https://github.com/grafana/loki/pull/3024) **jeschkies**: Collapse multiline logs based on a start line. +* [3015](https://github.com/grafana/loki/pull/3015) **cyriltovena**: Add more information about why a tailer would stop. +* [2996](https://github.com/grafana/loki/pull/2996) **cyriltovena**: Asynchronous Promtail stages +* [2898](https://github.com/grafana/loki/pull/2898) **kavirajk**: fix(docker-driver): Propagate promtail's `client.Stop` properly + +#### Logcli + +* [3325](https://github.com/grafana/loki/pull/3325) **cyriltovena**: Fixes step encoding in logcli. +* [3271](https://github.com/grafana/loki/pull/3271) **chancez**: Refactor logcli Client interface to use time objects for LiveTailQueryConn +* [3270](https://github.com/grafana/loki/pull/3270) **chancez**: logcli: Fix handling of logcli query using --since/--from and --tail +* [3229](https://github.com/grafana/loki/pull/3229) **dethi**: logcli: support --include-label when not using --tail + + +#### Jsonnet + +* [3447](https://github.com/grafana/loki/pull/3447) **owen-d**: Use better memory metric on operational dashboard +* [3439](https://github.com/grafana/loki/pull/3439) **owen-d**: simplifies jsonnet sharding +* [3357](https://github.com/grafana/loki/pull/3357) **owen-d**: Libsonnet/better sharding parallelism defaults +* [3356](https://github.com/grafana/loki/pull/3356) **owen-d**: removes sharding queue math after global concurrency PR +* [3329](https://github.com/grafana/loki/pull/3329) **sandeepsukhani**: fix config for statefulset rulers when using boltdb-shipper +* [3297](https://github.com/grafana/loki/pull/3297) **owen-d**: adds stateful ruler clause for boltdb shipper jsonnet +* [3254](https://github.com/grafana/loki/pull/3254) **hairyhenderson**: ksonnet: Remove invalid hostname from default promtail configuration +* [3181](https://github.com/grafana/loki/pull/3181) **owen-d**: remaining sts use parallel mgmt policy +* [3179](https://github.com/grafana/loki/pull/3179) **owen-d**: Ruler statefulsets +* [3156](https://github.com/grafana/loki/pull/3156) **slim-bean**: Jsonnet: Changes ingester PVC from 5Gi to 10Gi +* [3139](https://github.com/grafana/loki/pull/3139) **owen-d**: Less confusing jsonnet error message when using boltdb shipper defaults. +* [3079](https://github.com/grafana/loki/pull/3079) **rajatvig**: Fix ingester PVC data declaration to use configured value +* [3074](https://github.com/grafana/loki/pull/3074) **c0ffeec0der**: Ksonnet: Assign appropriate pvc size and class to compactor and ingester +* [3062](https://github.com/grafana/loki/pull/3062) **cyriltovena**: Remove regexes in the operational dashboard. +* [3014](https://github.com/grafana/loki/pull/3014) **owen-d**: loki wal libsonnet +* [3010](https://github.com/grafana/loki/pull/3010) **cyriltovena**: Fixes promtail mixin dashboard. + +#### fluentd + +* [3358](https://github.com/grafana/loki/pull/3358) **fpob**: Fix fluentd plugin when kubernetes labels were missing + +#### fluent bit + +* [3240](https://github.com/grafana/loki/pull/3240) **sbaier1**: fix fluent-bit output plugin generating invalid JSON + + +#### Docker Logging Driver + +* [3331](https://github.com/grafana/loki/pull/3331) **cyriltovena**: Add pprof endpoint to docker-driver. +* [3225](https://github.com/grafana/loki/pull/3225) **Le0tk0k**: (fix: cmd/docker-driver): Insert a space in the error message + +#### Docs + +* [3437](https://github.com/grafana/loki/pull/3437) **caleb15**: docs: add note about regex +* [3421](https://github.com/grafana/loki/pull/3421) **kavirajk**: doc(gcplog): Advanced log export filter example +* [3419](https://github.com/grafana/loki/pull/3419) **suitupalex**: docs: promtail: Fix typo w/ windows_events hyperlink. +* [3418](https://github.com/grafana/loki/pull/3418) **dannykopping**: Adding upgrade documentation for promtail pipeline_stages change +* [3385](https://github.com/grafana/loki/pull/3385) **paaacman**: Documentation: enable environment variable in configuration +* [3373](https://github.com/grafana/loki/pull/3373) **StMarian**: Documentation: Fix configuration description +* [3371](https://github.com/grafana/loki/pull/3371) **owen-d**: Distributor overview docs +* [3370](https://github.com/grafana/loki/pull/3370) **tkowalcz**: documentation: Add Tjahzi to the list of unofficial clients +* [3352](https://github.com/grafana/loki/pull/3352) **kavirajk**: Remove extra space between broken link +* [3351](https://github.com/grafana/loki/pull/3351) **kavirajk**: Add some operation details to gcplog doc +* [3316](https://github.com/grafana/loki/pull/3316) **kavirajk**: docs(fix): Make best practices docs look better +* [3292](https://github.com/grafana/loki/pull/3292) **wapmorgan**: Patch 2 - fix link to another documentation files +* [3265](https://github.com/grafana/loki/pull/3265) **sandeepsukhani**: Boltdb shipper doc fixes +* [3239](https://github.com/grafana/loki/pull/3239) **owen-d**: updates tanka installation docs +* [3235](https://github.com/grafana/loki/pull/3235) **scoof**: docs: point to latest release for docker installation +* [3220](https://github.com/grafana/loki/pull/3220) **liguozhong**: [doc] fix err. "loki_frontend" is invalid +* [3212](https://github.com/grafana/loki/pull/3212) **nvtkaszpir**: Fix: Update docs for logcli +* [3190](https://github.com/grafana/loki/pull/3190) **kavirajk**: doc(gcplog): Fix titles for Cloud provisioning for GCP logs +* [3165](https://github.com/grafana/loki/pull/3165) **liguozhong**: [doc] fix:querier do not handle "/flush" api +* [3164](https://github.com/grafana/loki/pull/3164) **owen-d**: updates alerting docs post 2.0 +* [3162](https://github.com/grafana/loki/pull/3162) **huikang**: Doc: Add missing wal in configuration +* [3148](https://github.com/grafana/loki/pull/3148) **huikang**: Doc: add missing type supported by table manager +* [3147](https://github.com/grafana/loki/pull/3147) **marionxue**: Markdown Code highlighting +* [3138](https://github.com/grafana/loki/pull/3138) **jeschkies**: Give another example for multiline. +* [3128](https://github.com/grafana/loki/pull/3128) **cyriltovena**: Fixes LogQL documentation links. +* [3124](https://github.com/grafana/loki/pull/3124) **wujie1993**: fix time duration unit +* [3123](https://github.com/grafana/loki/pull/3123) **scoren-gl**: Update getting-in-touch.md +* [3115](https://github.com/grafana/loki/pull/3115) **valmack**: Docs: Include instruction to enable variable expansion +* [3109](https://github.com/grafana/loki/pull/3109) **nileshcs**: Documentation fix for downstream_url +* [3102](https://github.com/grafana/loki/pull/3102) **slim-bean**: Docs: Changelog: fix indentation and add helm repo url +* [3094](https://github.com/grafana/loki/pull/3094) **benjaminhuo**: Fix storage guide links +* [3088](https://github.com/grafana/loki/pull/3088) **cyriltovena**: Small fixes for the documentation. +* [3084](https://github.com/grafana/loki/pull/3084) **ilpianista**: Update reference to old helm chart repo +* [3078](https://github.com/grafana/loki/pull/3078) **kavirajk**: mention the use of `config.expand-env` flag in the doc. +* [3049](https://github.com/grafana/loki/pull/3049) **vitalets**: [Docs] Clarify docker-driver configuration options +* [3039](https://github.com/grafana/loki/pull/3039) **jdbaldry**: doc: logql formatting fixes +* [3035](https://github.com/grafana/loki/pull/3035) **unguiculus**: Fix multiline docs +* [3033](https://github.com/grafana/loki/pull/3033) **RichiH**: docs: Create ADOPTERS.md +* [3032](https://github.com/grafana/loki/pull/3032) **oddlittlebird**: Docs: Update _index.md +* [3029](https://github.com/grafana/loki/pull/3029) **jeschkies**: Correct `multiline` documentation. +* [3027](https://github.com/grafana/loki/pull/3027) **nop33**: Fix docs header inconsistency +* [3026](https://github.com/grafana/loki/pull/3026) **owen-d**: wal docs +* [3017](https://github.com/grafana/loki/pull/3017) **jdbaldry**: doc: Cleanup formatting +* [3009](https://github.com/grafana/loki/pull/3009) **jdbaldry**: doc: Fix query-frontend typo +* [3002](https://github.com/grafana/loki/pull/3002) **keyolk**: Fix typo +* [2991](https://github.com/grafana/loki/pull/2991) **jontg**: Documentation: Add a missing field to the extended config s3 example +* [2956](https://github.com/grafana/loki/pull/2956) **owen-d**: Updates chunkenc doc for V3 + +#### Build + +* [3412](https://github.com/grafana/loki/pull/3412) **rfratto**: Remove unneeded prune-ci-tags drone job +* [3390](https://github.com/grafana/loki/pull/3390) **wardbekker**: Don't auto-include pod labels as loki labels as a sane default +* [3321](https://github.com/grafana/loki/pull/3321) **owen-d**: defaults promtail to 2.1.0 in install script +* [3277](https://github.com/grafana/loki/pull/3277) **kavirajk**: Add step to version Loki docs during public release process. +* [3243](https://github.com/grafana/loki/pull/3243) **chancez**: dist: Build promtail for windows/386 to support 32 bit windows hosts +* [3206](https://github.com/grafana/loki/pull/3206) **kavirajk**: Terraform script to automate GCP provisioning for gcplog +* [3149](https://github.com/grafana/loki/pull/3149) **jlosito**: Allow dependabot to keep github actions up-to-date +* [3072](https://github.com/grafana/loki/pull/3072) **slim-bean**: Helm: Disable CI +* [3031](https://github.com/grafana/loki/pull/3031) **AdamKorcz**: Testing: Introduced continuous fuzzing +* [3006](https://github.com/grafana/loki/pull/3006) **huikang**: Fix the docker image version in compose deployment + + +#### Tooling + +* [3377](https://github.com/grafana/loki/pull/3377) **slim-bean**: Tooling: Update chunks-inspect to understand the new chunk format as well as new compression algorithms +* [3151](https://github.com/grafana/loki/pull/3151) **slim-bean**: Loki migrate-tool + + +### Notes + +This release was created from revision 8012362674568379a3871ff8c4a2bfd1ddba7ad1 (Which was PR 3460) + +### Dependencies + +* Go Version: 1.15.3 +* Cortex Version: 7dac81171c665be071bd167becd1f55528a9db32 + + ## 2.1.0 (2020/12/23) Happy Holidays from the Loki team! Please enjoy a new Loki release to welcome in the New Year! diff --git a/docs/sources/installation/docker.md b/docs/sources/installation/docker.md index e619604cb82c..73b373161f5c 100644 --- a/docs/sources/installation/docker.md +++ b/docs/sources/installation/docker.md @@ -18,10 +18,10 @@ For production, we recommend installing with Tanka or Helm. Copy and paste the commands below into your command line. ```bash -wget https://raw.githubusercontent.com/grafana/loki/v2.1.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml -docker run -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:2.1.0 -config.file=/mnt/config/loki-config.yaml -wget https://raw.githubusercontent.com/grafana/loki/v2.1.0/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml -docker run -v $(pwd):/mnt/config -v /var/log:/var/log grafana/promtail:2.1.0 -config.file=/mnt/config/promtail-config.yaml +wget https://raw.githubusercontent.com/grafana/loki/v2.2.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml +docker run -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:2.2.0 -config.file=/mnt/config/loki-config.yaml +wget https://raw.githubusercontent.com/grafana/loki/v2.2.0/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml +docker run -v $(pwd):/mnt/config -v /var/log:/var/log grafana/promtail:2.2.0 -config.file=/mnt/config/promtail-config.yaml ``` When finished, `loki-config.yaml` and `promtail-config.yaml` are downloaded in the directory you chose. Docker containers are running Loki and Promtail using those config files. @@ -36,10 +36,10 @@ Copy and paste the commands below into your terminal. Note that you will need to ```bash cd "" -wget https://raw.githubusercontent.com/grafana/loki/v2.1.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml -docker run -v :/mnt/config -p 3100:3100 grafana/loki:2.1.0 --config.file=/mnt/config/loki-config.yaml -wget https://raw.githubusercontent.com/grafana/loki/v2.1.0/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml -docker run -v :/mnt/config -v /var/log:/var/log grafana/promtail:2.1.0 --config.file=/mnt/config/promtail-config.yaml +wget https://raw.githubusercontent.com/grafana/loki/v2.2.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml +docker run -v :/mnt/config -p 3100:3100 grafana/loki:2.2.0 --config.file=/mnt/config/loki-config.yaml +wget https://raw.githubusercontent.com/grafana/loki/v2.2.0/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml +docker run -v :/mnt/config -v /var/log:/var/log grafana/promtail:2.2.0 --config.file=/mnt/config/promtail-config.yaml ``` When finished, `loki-config.yaml` and `promtail-config.yaml` are downloaded in the directory you chose. Docker containers are running Loki and Promtail using those config files. @@ -51,6 +51,6 @@ Navigate to http://localhost:3100/metrics to view the output. Run the following commands in your command line. They work for Windows or Linux systems. ```bash -wget https://raw.githubusercontent.com/grafana/loki/v2.1.0/production/docker-compose.yaml -O docker-compose.yaml +wget https://raw.githubusercontent.com/grafana/loki/v2.2.0/production/docker-compose.yaml -O docker-compose.yaml docker-compose -f docker-compose.yaml up ``` diff --git a/docs/sources/upgrading/_index.md b/docs/sources/upgrading/_index.md index edf69d8f47f8..5d91ae399750 100644 --- a/docs/sources/upgrading/_index.md +++ b/docs/sources/upgrading/_index.md @@ -19,12 +19,29 @@ If possible try to stay current and do sequential updates. If you want to skip v -_add changes here which are unreleased_ -### Promtail config changes +## 2.2.0 -In [this PR](https://github.com/grafana/loki/pull/3404), we reverted a bug that caused `scrape_configs` entries without a -`pipeline_stages` definition to default to the `docker` pipeline stage. +### Loki -If any of your `scrape_configs` are missing this definition, you should add the following to maintain this behaviour: +**Be sure to upgrade to 2.0 or 2.1 BEFORE upgrading to 2.2** + +In Loki 2.2 we changed the internal version of our chunk format from v2 to v3, this is a transparent change and is only relevant if you every try to _downgrade_ a Loki installation. We incorporated the code to read v3 chunks in 2.0.1 and 2.1, as well as 2.2 and any future releases. + +**If you upgrade to 2.2+ any chunks created can only be read by 2.0.1, 2.1 and 2.2+** + +This makes it important to first upgrade to 2.0, 2.0.1, or 2.1 **before** upgrading to 2.2 so that if you need to rollback for any reason you can do so easily. + +**Note:** 2.0 and 2.0.1 are identical in every aspect except 2.0.1 contains the code necessary to read the v3 chunk format. Therefor if you are on 2.0 and ugrade to 2.2, if you want to rollback, you must rollback to 2.0.1. + +### Promtail + +For 2.0 we eliminated the long deprecated `entry_parser` configuration in Promtail configs, however in doing so we introduced a very confusing and erroneous default behavior: + +If you did not specify a `pipeline_stages` entry you would be provided with a default which included the `docker` pipeline stage. This can lead to some very confusing results. + +In [3404](https://github.com/grafana/loki/pull/3404), we corrected this behavior + +**If you are using docker, and any of your `scrape_configs` are missing a `pipeline_stages` definition**, you should add the following to obtain the correct behaviour: ```yaml pipeline_stages: From a27c79913c43d02b62a4b2f479bd77e9aa6b3617 Mon Sep 17 00:00:00 2001 From: Edward Welch Date: Thu, 11 Mar 2021 08:19:30 -0500 Subject: [PATCH 2/9] add upgrade note about fix to query_frontend queueing. --- CHANGELOG.md | 2 +- docs/sources/upgrading/_index.md | 18 ++++++++++++++++++ 2 files changed, 19 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 27f4acdb38ec..9b9b3c588bba 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,7 +18,7 @@ Thanks to everyone for another exciting Loki release!! Please read the [Upgrade Guide](https://github.com/grafana/loki/blob/master/docs/sources/upgrading/_index.md#220) before upgrading for a smooth experience. -TL;DR Loki 2.2 changes the internal chunk format which limits what versions you can downgrade to, and we fixed a mistake related `scrape_configs` which do not have a `pipeline_stages` defined. If you have any Promtail `scrape_configs` which do not specify `pipeline_stages` you should go read the upgrade notes! +TL;DR Loki 2.2 changes the internal chunk format which limits what versions you can downgrade to, a bug in how many queries were allowed to be scheduled per tenant was fixed which might affect your `max_query_parallelism` and `max_outstanding_per_tenant` settings, and we fixed a mistake related `scrape_configs` which do not have a `pipeline_stages` defined. If you have any Promtail `scrape_configs` which do not specify `pipeline_stages` you should go read the upgrade notes! ### All Changes diff --git a/docs/sources/upgrading/_index.md b/docs/sources/upgrading/_index.md index 5d91ae399750..e981ecb189a5 100644 --- a/docs/sources/upgrading/_index.md +++ b/docs/sources/upgrading/_index.md @@ -33,6 +33,24 @@ This makes it important to first upgrade to 2.0, 2.0.1, or 2.1 **before** upgrad **Note:** 2.0 and 2.0.1 are identical in every aspect except 2.0.1 contains the code necessary to read the v3 chunk format. Therefor if you are on 2.0 and ugrade to 2.2, if you want to rollback, you must rollback to 2.0.1. +### Loki Config + +**Read this if you use the query-frontend and have `sharded_queries_enabled: true`** + +We discovered query scheduling related to sharded queries over long time ranges could lead to unfair work scheduling by one single query in the per tenant work queue. + +The `max_query_parallelism` setting is designed to limit how many split and sharded units of 'work' for a single query are allowed to be put into the per tenant work queue at one time. The previous behavior would split the query by time using the `split_queries_by_interval` and compare this value to `max_query_parallelism` when filling the queue, however with sharding enabled, every split was then sharded into 16 additional units of work after the `max_query_parallelism` limit was applied. + +In 2.2 we changed this behavior to apply the `max_query_parallelism` after splitting _and_ sharding a query resulting a more fair and expected queue scheduling per query. + +**What this means** Loki will be putting much less work into the work queue per query if you are using the query frontend and have sharding_queries_enabled (which you should). **You may need to increase your `max_query_parallelism` setting if you are noticing slower query performance** In practice, you may not see a difference unless you were running a cluster with a LOT of queriers or queriers with a very high `parallelism` frontend_worker setting. + +You could consider multiplying your current `max_query_parallelism` setting by 16 to obtain the previous behavior, though in practice we suspect few people would really want it this high unless you have a significant querier worker pool. + +**Also be aware to make sure `max_outsdanting_per_tenant` is always greater than `max_query_parallelism` or large queries will automatically fail with a 429 back to the user.** + + + ### Promtail For 2.0 we eliminated the long deprecated `entry_parser` configuration in Promtail configs, however in doing so we introduced a very confusing and erroneous default behavior: From 3b30483b772f8b07801ab26b666e825cfe71b2db Mon Sep 17 00:00:00 2001 From: Aditya C S Date: Wed, 17 Mar 2021 18:34:53 +0530 Subject: [PATCH 3/9] Support labelallow stage in Promtail (#3468) * add labelallow stage * added docs * add docs * fix lint * update stage name in _index.md (cherry picked from commit 0f5fceef8f6f18f4c341f07cf7ae3d18e9534822) --- .../sources/clients/promtail/stages/_index.md | 1 + .../clients/promtail/stages/labelallow.md | 41 +++++++++++ .../clients/promtail/stages/labeldrop.md | 2 +- pkg/logentry/stages/labelallow.go | 65 +++++++++++++++++ pkg/logentry/stages/labelallow_test.go | 72 +++++++++++++++++++ pkg/logentry/stages/stage.go | 40 ++++++----- 6 files changed, 203 insertions(+), 18 deletions(-) create mode 100644 docs/sources/clients/promtail/stages/labelallow.md create mode 100644 pkg/logentry/stages/labelallow.go create mode 100644 pkg/logentry/stages/labelallow_test.go diff --git a/docs/sources/clients/promtail/stages/_index.md b/docs/sources/clients/promtail/stages/_index.md index 5b5e2b833a6d..15969926862e 100644 --- a/docs/sources/clients/promtail/stages/_index.md +++ b/docs/sources/clients/promtail/stages/_index.md @@ -24,6 +24,7 @@ Action stages: - [timestamp](timestamp/): Set the timestamp value for the log entry. - [output](output/): Set the log line text. - [labeldrop](labeldrop/): Drop label set for the log entry. + - [labelallow](labelallow/): Allow label set for the log entry. - [labels](labels/): Update the label set for the log entry. - [metrics](metrics/): Calculate metrics based on extracted data. - [tenant](tenant/): Set the tenant ID value to use for the log entry. diff --git a/docs/sources/clients/promtail/stages/labelallow.md b/docs/sources/clients/promtail/stages/labelallow.md new file mode 100644 index 000000000000..46d07d208278 --- /dev/null +++ b/docs/sources/clients/promtail/stages/labelallow.md @@ -0,0 +1,41 @@ +--- +title: labelallow +--- +# `labelallow` stage + +The labelallow stage is an action stage that allows only the provided labels +to be included in the label set that is sent to Loki with the log entry. + +## Schema + +```yaml +labelallow: + - [] + ... +``` + +### Examples + +For the given pipeline: + +```yaml +kubernetes_sd_configs: + - role: pod +pipeline_stages: +- docker: {} +- labelallow: + - kubernetes_pod_name + - kubernetes_container_name +``` + +Given the following incoming labels: + +- `kubernetes_pod_name`: `"loki-pqrs"` +- `kubernetes_container_name`: `"loki"` +- `kubernetes_pod_template_hash`: `"79f5db67b"` +- `kubernetes_controller_revision_hash`: `"774858987d"` + +Only the below labels would be sent to `loki` + +- `kubernetes_pod_name`: `"loki-pqrs"` +- `contaikubernetes_container_namener`: `"loki"` diff --git a/docs/sources/clients/promtail/stages/labeldrop.md b/docs/sources/clients/promtail/stages/labeldrop.md index 6559fb558c7e..e5b08be2e142 100644 --- a/docs/sources/clients/promtail/stages/labeldrop.md +++ b/docs/sources/clients/promtail/stages/labeldrop.md @@ -3,7 +3,7 @@ title: labeldrop --- # `labeldrop` stage -The labeldrop stage is an action stage that takes drops labels from +The labeldrop stage is an action stage that drops labels from the label set that is sent to Loki with the log entry. ## Schema diff --git a/pkg/logentry/stages/labelallow.go b/pkg/logentry/stages/labelallow.go new file mode 100644 index 000000000000..8ab51c4aa659 --- /dev/null +++ b/pkg/logentry/stages/labelallow.go @@ -0,0 +1,65 @@ +package stages + +import ( + "time" + + "github.com/mitchellh/mapstructure" + "github.com/pkg/errors" + "github.com/prometheus/common/model" +) + +const ( + // ErrEmptyLabelAllowStageConfig error returned if config is empty + ErrEmptyLabelAllowStageConfig = "labelallow stage config cannot be empty" +) + +// labelallowConfig is a slice of labels to be included +type LabelAllowConfig []string + +func validateLabelAllowConfig(c LabelAllowConfig) error { + if c == nil || len(c) < 1 { + return errors.New(ErrEmptyLabelAllowStageConfig) + } + + return nil +} + +func newLabelAllowStage(configs interface{}) (Stage, error) { + cfgs := &LabelAllowConfig{} + err := mapstructure.Decode(configs, cfgs) + if err != nil { + return nil, err + } + + err = validateLabelAllowConfig(*cfgs) + if err != nil { + return nil, err + } + + labelMap := make(map[string]struct{}) + for _, label := range *cfgs { + labelMap[label] = struct{}{} + } + + return toStage(&labelAllowStage{ + labels: labelMap, + }), nil +} + +type labelAllowStage struct { + labels map[string]struct{} +} + +// Process implements Stage +func (l *labelAllowStage) Process(labels model.LabelSet, extracted map[string]interface{}, t *time.Time, entry *string) { + for label := range labels { + if _, ok := l.labels[string(label)]; !ok { + delete(labels, label) + } + } +} + +// Name implements Stage +func (l *labelAllowStage) Name() string { + return StageTypeLabelAllow +} diff --git a/pkg/logentry/stages/labelallow_test.go b/pkg/logentry/stages/labelallow_test.go new file mode 100644 index 000000000000..2c982db29544 --- /dev/null +++ b/pkg/logentry/stages/labelallow_test.go @@ -0,0 +1,72 @@ +package stages + +import ( + "testing" + "time" + + util_log "github.com/cortexproject/cortex/pkg/util/log" + "github.com/prometheus/common/model" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + ww "github.com/weaveworks/common/server" +) + +func Test_addLabelStage_Process(t *testing.T) { + // Enable debug logging + cfg := &ww.Config{} + require.Nil(t, cfg.LogLevel.Set("debug")) + util_log.InitLogger(cfg) + Debug = true + + tests := []struct { + name string + config *LabelAllowConfig + inputLabels model.LabelSet + expectedLabels model.LabelSet + }{ + { + name: "allow single label", + config: &LabelAllowConfig{"testLabel1"}, + inputLabels: model.LabelSet{ + "testLabel1": "testValue", + "testLabel2": "testValue", + }, + expectedLabels: model.LabelSet{ + "testLabel1": "testValue", + }, + }, + { + name: "allow multiple labels", + config: &LabelAllowConfig{"testLabel1", "testLabel2"}, + inputLabels: model.LabelSet{ + "testLabel1": "testValue", + "testLabel2": "testValue", + "testLabel3": "testValue", + }, + expectedLabels: model.LabelSet{ + "testLabel1": "testValue", + "testLabel2": "testValue", + }, + }, + { + name: "allow non-existing label", + config: &LabelAllowConfig{"foobar"}, + inputLabels: model.LabelSet{ + "testLabel1": "testValue", + "testLabel2": "testValue", + }, + expectedLabels: model.LabelSet{}, + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + st, err := newLabelAllowStage(test.config) + if err != nil { + t.Fatal(err) + } + out := processEntries(st, newEntry(nil, test.inputLabels, "", time.Now()))[0] + assert.Equal(t, test.expectedLabels, out.Labels) + }) + } +} diff --git a/pkg/logentry/stages/stage.go b/pkg/logentry/stages/stage.go index c3e77f066560..d89c26c2b031 100644 --- a/pkg/logentry/stages/stage.go +++ b/pkg/logentry/stages/stage.go @@ -12,23 +12,24 @@ import ( ) const ( - StageTypeJSON = "json" - StageTypeRegex = "regex" - StageTypeReplace = "replace" - StageTypeMetric = "metrics" - StageTypeLabel = "labels" - StageTypeLabelDrop = "labeldrop" - StageTypeTimestamp = "timestamp" - StageTypeOutput = "output" - StageTypeDocker = "docker" - StageTypeCRI = "cri" - StageTypeMatch = "match" - StageTypeTemplate = "template" - StageTypePipeline = "pipeline" - StageTypeTenant = "tenant" - StageTypeDrop = "drop" - StageTypeMultiline = "multiline" - StageTypePack = "pack" + StageTypeJSON = "json" + StageTypeRegex = "regex" + StageTypeReplace = "replace" + StageTypeMetric = "metrics" + StageTypeLabel = "labels" + StageTypeLabelDrop = "labeldrop" + StageTypeTimestamp = "timestamp" + StageTypeOutput = "output" + StageTypeDocker = "docker" + StageTypeCRI = "cri" + StageTypeMatch = "match" + StageTypeTemplate = "template" + StageTypePipeline = "pipeline" + StageTypeTenant = "tenant" + StageTypeDrop = "drop" + StageTypeMultiline = "multiline" + StageTypePack = "pack" + StageTypeLabelAllow = "labelallow" ) // Processor takes an existing set of labels, timestamp and log entry and returns either a possibly mutated @@ -151,6 +152,11 @@ func New(logger log.Logger, jobName *string, stageType string, if err != nil { return nil, err } + case StageTypeLabelAllow: + s, err = newLabelAllowStage(cfg) + if err != nil { + return nil, err + } default: return nil, errors.Errorf("Unknown stage type: %s", stageType) } From 606755be520d6561f8fa9ede70f5546a1b7009f7 Mon Sep 17 00:00:00 2001 From: Cyril Tovena Date: Wed, 17 Mar 2021 10:19:42 -0400 Subject: [PATCH 4/9] Fixes a bug where unpack would mutate log line. (#3502) * Fixes a bug where unpack would mutate log line. Signed-off-by: Cyril Tovena * add reset. Signed-off-by: Cyril Tovena (cherry picked from commit 6148794b9bcd18b67b26254703ce9f75fd0d070d) --- pkg/logql/log/parser.go | 3 +-- pkg/logql/log/parser_test.go | 3 +++ 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/pkg/logql/log/parser.go b/pkg/logql/log/parser.go index 64136f0e6276..cf23b01f9946 100644 --- a/pkg/logql/log/parser.go +++ b/pkg/logql/log/parser.go @@ -358,11 +358,10 @@ func (u *UnpackParser) unpack(it *jsoniter.Iterator, entry []byte, lbs *LabelsBu case jsoniter.StringValue: // we only unpack map[string]string. Anything else is skipped. if field == PackedEntryKey { - s := iter.ReadString() // todo(ctovena): we should just reslice the original line since the property is contiguous // but jsoniter doesn't allow us to do this right now. // https://github.com/buger/jsonparser might do a better job at this. - entry = append(entry[:0], []byte(s)...) + entry = []byte(iter.ReadString()) isPacked = true return true } diff --git a/pkg/logql/log/parser_test.go b/pkg/logql/log/parser_test.go index c2055eb60fbc..6e57604c9b93 100644 --- a/pkg/logql/log/parser_test.go +++ b/pkg/logql/log/parser_test.go @@ -721,10 +721,13 @@ func Test_unpackParser_Parse(t *testing.T) { t.Run(tt.name, func(t *testing.T) { b := NewBaseLabelsBuilder().ForLabels(tt.lbs, tt.lbs.Hash()) b.Reset() + copy := string(tt.line) l, _ := j.Process(tt.line, b) sort.Sort(tt.wantLbs) require.Equal(t, tt.wantLbs, b.Labels()) require.Equal(t, tt.wantLine, l) + require.Equal(t, string(tt.wantLine), string(l)) + require.Equal(t, copy, string(tt.line), "the original log line should not be mutated") }) } } From 2c5d79b7f98bd8a3175e66163979afa6628fae66 Mon Sep 17 00:00:00 2001 From: Cyril Tovena Date: Thu, 25 Mar 2021 14:21:11 -0400 Subject: [PATCH 5/9] Support for single step metric query. (#3540) When splitting correctly steps in the frontend, some queries might be of a single step, e.g start == end. This PR add supports for those queries, which are already supported by Cortex and Prometheus, but for some reason not Loki. Signed-off-by: Cyril Tovena (cherry picked from commit 29a3240e17b9d76ef5f4e5556d145eddb7355a83) --- pkg/loghttp/query.go | 2 +- pkg/loghttp/query_test.go | 78 +++++++++++++++++++++++---------------- 2 files changed, 48 insertions(+), 32 deletions(-) diff --git a/pkg/loghttp/query.go b/pkg/loghttp/query.go index f343e531f4e3..f12180eab8bd 100644 --- a/pkg/loghttp/query.go +++ b/pkg/loghttp/query.go @@ -222,7 +222,7 @@ func ParseRangeQuery(r *http.Request) (*RangeQuery, error) { return nil, err } - if result.End.Before(result.Start) || result.Start.Equal(result.End) { + if result.End.Before(result.Start) { return nil, errEndBeforeStart } diff --git a/pkg/loghttp/query_test.go b/pkg/loghttp/query_test.go index 54b180378ab6..1c4a9dc28fee 100644 --- a/pkg/loghttp/query_test.go +++ b/pkg/loghttp/query_test.go @@ -24,29 +24,39 @@ func TestParseRangeQuery(t *testing.T) { {"bad start", &http.Request{URL: mustParseURL(`?query={foo="bar"}&start=t`)}, nil, true}, {"bad end", &http.Request{URL: mustParseURL(`?query={foo="bar"}&end=t`)}, nil, true}, {"end before start", &http.Request{URL: mustParseURL(`?query={foo="bar"}&start=2016-06-10T21:42:24.760738998Z&end=2015-06-10T21:42:24.760738998Z`)}, nil, true}, - {"end equal start", &http.Request{URL: mustParseURL(`?query={foo="bar"}&start=2016-06-10T21:42:24.760738998Z&end=2016-06-10T21:42:24.760738998Z`)}, nil, true}, {"bad limit", &http.Request{URL: mustParseURL(`?query={foo="bar"}&start=2016-06-10T21:42:24.760738998Z&end=2017-06-10T21:42:24.760738998Z&limit=h`)}, nil, true}, - {"bad direction", + { + "bad direction", &http.Request{ URL: mustParseURL(`?query={foo="bar"}&start=2016-06-10T21:42:24.760738998Z&end=2017-06-10T21:42:24.760738998Z&limit=100&direction=fw`), - }, nil, true}, - {"bad step", + }, nil, true, + }, + { + "bad step", &http.Request{ URL: mustParseURL(`?query={foo="bar"}&start=2016-06-10T21:42:24.760738998Z&end=2017-06-10T21:42:24.760738998Z&limit=100&direction=FORWARD&step=h`), - }, nil, true}, - {"negative step", + }, nil, true, + }, + { + "negative step", &http.Request{ URL: mustParseURL(`?query={foo="bar"}&start=2016-06-10T21:42:24.760738998Z&end=2017-06-10T21:42:24.760738998Z&limit=100&direction=BACKWARD&step=-1`), - }, nil, true}, - {"too small step", + }, nil, true, + }, + { + "too small step", &http.Request{ URL: mustParseURL(`?query={foo="bar"}&start=2016-06-10T21:42:24.760738998Z&end=2017-06-10T21:42:24.760738998Z&limit=100&direction=BACKWARD&step=1`), - }, nil, true}, - {"negative interval", + }, nil, true, + }, + { + "negative interval", &http.Request{ URL: mustParseURL(`?query={foo="bar"}&start=2016-06-10T21:42:24.760738998Z&end=2017-06-10T21:42:24.760738998Z&limit=100&direction=BACKWARD&step=1,interval=-1`), - }, nil, true}, - {"good", + }, nil, true, + }, + { + "good", &http.Request{ URL: mustParseURL(`?query={foo="bar"}&start=2017-06-10T21:42:24.760738998Z&end=2017-07-10T21:42:24.760738998Z&limit=1000&direction=BACKWARD&step=3600`), }, &RangeQuery{ @@ -56,7 +66,8 @@ func TestParseRangeQuery(t *testing.T) { Start: time.Date(2017, 06, 10, 21, 42, 24, 760738998, time.UTC), End: time.Date(2017, 07, 10, 21, 42, 24, 760738998, time.UTC), Limit: 1000, - }, false}, + }, false, + }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { @@ -76,7 +87,6 @@ func TestParseRangeQuery(t *testing.T) { } func TestParseInstantQuery(t *testing.T) { - tests := []struct { name string r *http.Request @@ -85,11 +95,14 @@ func TestParseInstantQuery(t *testing.T) { }{ {"bad time", &http.Request{URL: mustParseURL(`?query={foo="bar"}&time=t`)}, nil, true}, {"bad limit", &http.Request{URL: mustParseURL(`?query={foo="bar"}&time=2016-06-10T21:42:24.760738998Z&limit=h`)}, nil, true}, - {"bad direction", + { + "bad direction", &http.Request{ URL: mustParseURL(`?query={foo="bar"}&time=2016-06-10T21:42:24.760738998Z&limit=100&direction=fw`), - }, nil, true}, - {"good", + }, nil, true, + }, + { + "good", &http.Request{ URL: mustParseURL(`?query={foo="bar"}&time=2017-06-10T21:42:24.760738998Z&limit=1000&direction=BACKWARD`), }, &InstantQuery{ @@ -97,7 +110,8 @@ func TestParseInstantQuery(t *testing.T) { Direction: logproto.BACKWARD, Ts: time.Date(2017, 06, 10, 21, 42, 24, 760738998, time.UTC), Limit: 1000, - }, false}, + }, false, + }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { @@ -130,22 +144,24 @@ func TestStreams_ToProto(t *testing.T) { want []logproto.Stream }{ {"empty", nil, nil}, - {"some", []Stream{ - { - Labels: map[string]string{"foo": "bar"}, - Entries: []Entry{ - {Timestamp: time.Unix(0, 1), Line: "1"}, - {Timestamp: time.Unix(0, 2), Line: "2"}, + { + "some", + []Stream{ + { + Labels: map[string]string{"foo": "bar"}, + Entries: []Entry{ + {Timestamp: time.Unix(0, 1), Line: "1"}, + {Timestamp: time.Unix(0, 2), Line: "2"}, + }, }, - }, - { - Labels: map[string]string{"foo": "bar", "lvl": "error"}, - Entries: []Entry{ - {Timestamp: time.Unix(0, 3), Line: "3"}, - {Timestamp: time.Unix(0, 4), Line: "4"}, + { + Labels: map[string]string{"foo": "bar", "lvl": "error"}, + Entries: []Entry{ + {Timestamp: time.Unix(0, 3), Line: "3"}, + {Timestamp: time.Unix(0, 4), Line: "4"}, + }, }, }, - }, []logproto.Stream{ { Labels: `{foo="bar"}`, From b2aec4a4f3c4c688b0569dc318ba250160c629d2 Mon Sep 17 00:00:00 2001 From: Cyril Tovena Date: Mon, 29 Mar 2021 03:46:18 -0400 Subject: [PATCH 6/9] Fixes a bug in MatrixStepper when sharding queries. (#3550) * Fixes a bug in MatrixStepper when sharding queries. Since we split correctly metric queries, this bug has show itself. Basically we were not correctly stepping through time. We should always start from start, add the step until the start is after the end. For more read: https://www.robustperception.io/step-and-query_range Fixes #3541 Signed-off-by: Cyril Tovena * Fixes tests. Signed-off-by: Cyril Tovena * Fixes tests. for real. Signed-off-by: Cyril Tovena * Not easy to get those test working. Signed-off-by: Cyril Tovena (cherry picked from commit b11d2effe29d0ee497def7eef84634310563c602) --- pkg/logcli/query/query.go | 8 +------ pkg/logql/matrix.go | 2 +- pkg/logql/matrix_test.go | 48 +++++++++++++++++++++++++++++++++++---- pkg/logql/test_utils.go | 6 ++--- 4 files changed, 48 insertions(+), 16 deletions(-) diff --git a/pkg/logcli/query/query.go b/pkg/logcli/query/query.go index a3403c4b0298..efba440b86f5 100644 --- a/pkg/logcli/query/query.go +++ b/pkg/logcli/query/query.go @@ -58,7 +58,6 @@ type Query struct { // DoQuery executes the query and prints out the results func (q *Query) DoQuery(c client.Client, out output.LogOutput, statistics bool) { - if q.LocalConfig != "" { if err := q.DoLocalQuery(out, statistics, c.GetOrgID()); err != nil { log.Fatalf("Query failed: %+v", err) @@ -149,7 +148,6 @@ func (q *Query) DoQuery(c client.Client, out output.LogOutput, statistics bool) } } - } func (q *Query) printResult(value loghttp.ResultValue, out output.LogOutput, lastEntry []*loghttp.Entry) (int, []*loghttp.Entry) { @@ -172,7 +170,6 @@ func (q *Query) printResult(value loghttp.ResultValue, out output.LogOutput, las // DoLocalQuery executes the query against the local store using a Loki configuration file. func (q *Query) DoLocalQuery(out output.LogOutput, statistics bool, orgID string) error { - var conf loki.Config conf.RegisterFlags(flag.CommandLine) if q.LocalConfig == "" { @@ -255,7 +252,7 @@ func (q *Query) SetInstant(time time.Time) { } func (q *Query) isInstant() bool { - return q.Start == q.End + return q.Start == q.End && q.Step == 0 } func (q *Query) printStream(streams loghttp.Streams, out output.LogOutput, lastEntry []*loghttp.Entry) (int, []*loghttp.Entry) { @@ -369,7 +366,6 @@ func (q *Query) printMatrix(matrix loghttp.Matrix) { // it gives us more flexibility with regard to output types in the future. initially we are supporting just formatted json but eventually // we might add output options such as render to an image file on disk bytes, err := json.MarshalIndent(matrix, "", " ") - if err != nil { log.Fatalf("Error marshalling matrix: %v", err) } @@ -379,7 +375,6 @@ func (q *Query) printMatrix(matrix loghttp.Matrix) { func (q *Query) printVector(vector loghttp.Vector) { bytes, err := json.MarshalIndent(vector, "", " ") - if err != nil { log.Fatalf("Error marshalling vector: %v", err) } @@ -389,7 +384,6 @@ func (q *Query) printVector(vector loghttp.Vector) { func (q *Query) printScalar(scalar loghttp.Scalar) { bytes, err := json.MarshalIndent(scalar, "", " ") - if err != nil { log.Fatalf("Error marshalling scalar: %v", err) } diff --git a/pkg/logql/matrix.go b/pkg/logql/matrix.go index 4a1e40678148..e2feaeb6bb46 100644 --- a/pkg/logql/matrix.go +++ b/pkg/logql/matrix.go @@ -33,7 +33,7 @@ func NewMatrixStepper(start, end time.Time, step time.Duration, m promql.Matrix) func (m *MatrixStepper) Next() (bool, int64, promql.Vector) { m.ts = m.ts.Add(m.step) - if !m.ts.Before(m.end) { + if m.ts.After(m.end) { return false, 0, nil } diff --git a/pkg/logql/matrix_test.go b/pkg/logql/matrix_test.go index ddd0389c5c35..56a3d2a6d33b 100644 --- a/pkg/logql/matrix_test.go +++ b/pkg/logql/matrix_test.go @@ -20,7 +20,7 @@ func TestMatrixStepper(t *testing.T) { promql.Series{ Metric: labels.Labels{{Name: "foo", Value: "bar"}}, Points: []promql.Point{ - {T: start.UnixNano() / int64(step), V: 0}, + {T: start.UnixNano(), V: 0}, {T: start.Add(step).UnixNano() / int64(time.Millisecond), V: 1}, {T: start.Add(2*step).UnixNano() / int64(time.Millisecond), V: 2}, {T: start.Add(3*step).UnixNano() / int64(time.Millisecond), V: 3}, @@ -42,11 +42,11 @@ func TestMatrixStepper(t *testing.T) { expected := []promql.Vector{ { promql.Sample{ - Point: promql.Point{T: start.UnixNano() / int64(step), V: 0}, + Point: promql.Point{T: start.UnixNano(), V: 0}, Metric: labels.Labels{{Name: "foo", Value: "bar"}}, }, promql.Sample{ - Point: promql.Point{T: start.UnixNano() / int64(step), V: 0}, + Point: promql.Point{T: start.UnixNano(), V: 0}, Metric: labels.Labels{{Name: "bazz", Value: "buzz"}}, }, }, @@ -100,9 +100,19 @@ func TestMatrixStepper(t *testing.T) { Metric: labels.Labels{{Name: "bazz", Value: "buzz"}}, }, }, + { + promql.Sample{ + Point: promql.Point{T: start.Add(6*step).UnixNano() / int64(time.Millisecond), V: 0}, + Metric: labels.Labels{{Name: "foo", Value: "bar"}}, + }, + promql.Sample{ + Point: promql.Point{T: start.Add(6*step).UnixNano() / int64(time.Millisecond), V: 0}, + Metric: labels.Labels{{Name: "bazz", Value: "buzz"}}, + }, + }, } - for i := 0; i < int(end.Sub(start)/step); i++ { + for i := 0; i <= int(end.Sub(start)/step); i++ { ok, ts, vec := s.Next() require.Equal(t, ok, true) require.Equal(t, start.Add(step*time.Duration(i)).UnixNano()/int64(time.Millisecond), ts) @@ -113,3 +123,33 @@ func TestMatrixStepper(t *testing.T) { require.Equal(t, ok, false) } + +func Test_SingleStepMatrix(t *testing.T) { + var ( + start = time.Unix(0, 0) + end = time.Unix(0, 0) + step = time.Second + ) + + m := promql.Matrix{ + promql.Series{ + Metric: labels.Labels{}, + Points: []promql.Point{ + {T: start.UnixNano(), V: 10}, + }, + }, + } + + s := NewMatrixStepper(start, end, step, m) + + ok, ts, vec := s.Next() + require.True(t, ok) + require.Equal(t, start.UnixNano(), ts) + require.Equal(t, promql.Vector{promql.Sample{ + Point: promql.Point{T: start.UnixNano(), V: 10}, + Metric: labels.Labels{}, + }}, vec) + + ok, _, _ = s.Next() + require.False(t, ok) +} diff --git a/pkg/logql/test_utils.go b/pkg/logql/test_utils.go index c15970bcf726..cd610d33b0dc 100644 --- a/pkg/logql/test_utils.go +++ b/pkg/logql/test_utils.go @@ -201,7 +201,7 @@ outer: return iter.NewTimeRangedSampleIterator( iter.NewMultiSeriesIterator(ctx, filtered), req.Start.UnixNano(), - req.End.UnixNano(), + req.End.UnixNano()+1, ), nil } @@ -232,7 +232,6 @@ func (m MockDownstreamer) Downstream(ctx context.Context, queries []DownstreamQu results = append(results, res) } return results, nil - } // create nStreams of nEntries with labelNames each where each label value @@ -256,7 +255,7 @@ func randomStreams(nStreams, nEntries, nShards int, labelNames []string) (stream Value: fmt.Sprintf("%d", shard), }) } - for j := 0; j < nEntries; j++ { + for j := 0; j <= nEntries; j++ { stream.Entries = append(stream.Entries, logproto.Entry{ Timestamp: time.Unix(0, int64(j*int(time.Second))), Line: fmt.Sprintf("line number: %d", j), @@ -267,7 +266,6 @@ func randomStreams(nStreams, nEntries, nShards int, labelNames []string) (stream streams = append(streams, stream) } return streams - } func mustParseLabels(s string) labels.Labels { From e6bddfc3faf2febafa00038b6f41a51e3fa981e8 Mon Sep 17 00:00:00 2001 From: Cyril Tovena Date: Wed, 31 Mar 2021 16:40:23 -0400 Subject: [PATCH 7/9] Properly release the ticker in Loki client. (#3566) I think this was only impacting the docker driver who would start/stop for each new followed containers. My assumption is that this would slowly build up and eat CPU over time. Fixes #3319 I arrived to this conclusion since strace was showing a crazy amount of futex syscall and nothing else seems to be the cause. :pray: Signed-off-by: Cyril Tovena (cherry picked from commit ecca5d345c7c3c6d06572d019d8dd5d7cbefc46c) --- pkg/promtail/client/client.go | 1 + 1 file changed, 1 insertion(+) diff --git a/pkg/promtail/client/client.go b/pkg/promtail/client/client.go index 19c4a2da48a2..2dbe134f1d8c 100644 --- a/pkg/promtail/client/client.go +++ b/pkg/promtail/client/client.go @@ -217,6 +217,7 @@ func (c *client) run() { maxWaitCheck := time.NewTicker(maxWaitCheckFrequency) defer func() { + maxWaitCheck.Stop() // Send all pending batches for tenantID, batch := range batches { c.sendBatch(tenantID, batch) From a1a9e6b72e5b0ebb64e6822d7e449566f67b456a Mon Sep 17 00:00:00 2001 From: Cyril Tovena Date: Mon, 5 Apr 2021 20:17:48 -0400 Subject: [PATCH 8/9] Fixes a race when using specific tenant and multi-client. (#3573) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Fixes a race when using specific tenant and multi-client. This was because each client would try to mutate the original set of labels. Instead of cloning the map which can be expensive, I created a little helper that generate the string from the labelset without a set of label name. Fixes #3571 I also added a test to see if this was reproducible and it was indeed: ``` ~/go/src/github.com/grafana/loki master* ❯ go test -timeout 30s -tags dev,gofuzz -race -run ^TestMultiClient_Handle_Race$ github.com/grafana/loki/pkg/promtail/client -v -count=1 -timeout=0s === RUN TestMultiClient_Handle_Race ================== WARNING: DATA RACE Read at 0x00c0000b77d0 by goroutine 22: runtime.mapaccess2_faststr() /usr/local/Cellar/go/1.16.2/libexec/src/runtime/map_faststr.go:107 +0x0 github.com/grafana/loki/pkg/promtail/client.(*client).getTenantID() /Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:376 +0xcb github.com/grafana/loki/pkg/promtail/client.(*client).processEntry() /Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:407 +0x9a github.com/grafana/loki/pkg/promtail/client.(*client).run() /Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:235 +0x36a Previous write at 0x00c0000b77d0 by goroutine 21: runtime.mapdelete_faststr() /usr/local/Cellar/go/1.16.2/libexec/src/runtime/map_faststr.go:297 +0x0 github.com/grafana/loki/pkg/promtail/client.(*client).processEntry() /Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:408 +0x144 github.com/grafana/loki/pkg/promtail/client.(*client).run() /Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:235 +0x36a Goroutine 22 (running) created at: github.com/grafana/loki/pkg/promtail/client.New() /Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:198 +0x7dc github.com/grafana/loki/pkg/promtail/client.TestMultiClient_Handle_Race() /Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/multi_test.go:126 +0x2c7 testing.tRunner() /usr/local/Cellar/go/1.16.2/libexec/src/testing/testing.go:1194 +0x202 Goroutine 21 (running) created at: github.com/grafana/loki/pkg/promtail/client.New() /Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:198 +0x7dc github.com/grafana/loki/pkg/promtail/client.TestMultiClient_Handle_Race() /Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/multi_test.go:124 +0x187 testing.tRunner() /usr/local/Cellar/go/1.16.2/libexec/src/testing/testing.go:1194 +0x202 ================== testing.go:1093: race detected during execution of test --- FAIL: TestMultiClient_Handle_Race (0.00s) === CONT testing.go:1093: race detected during execution of test FAIL FAIL github.com/grafana/loki/pkg/promtail/client 0.022s FAIL ``` Signed-off-by: Cyril Tovena * don't overload variable name Co-authored-by: Edward Welch (cherry picked from commit 0107a117691f936459032a52eb035d2f235dbce9) --- pkg/promtail/client/batch.go | 22 +++++++++++++++++++++- pkg/promtail/client/client.go | 1 - pkg/promtail/client/multi_test.go | 22 ++++++++++++++++++++++ 3 files changed, 43 insertions(+), 2 deletions(-) diff --git a/pkg/promtail/client/batch.go b/pkg/promtail/client/batch.go index 16bb23ebbe92..16c5d684a36c 100644 --- a/pkg/promtail/client/batch.go +++ b/pkg/promtail/client/batch.go @@ -1,10 +1,14 @@ package client import ( + "fmt" + "sort" + "strings" "time" "github.com/gogo/protobuf/proto" "github.com/golang/snappy" + "github.com/prometheus/common/model" "github.com/grafana/loki/pkg/logproto" "github.com/grafana/loki/pkg/promtail/api" @@ -40,7 +44,7 @@ func (b *batch) add(entry api.Entry) { b.bytes += len(entry.Line) // Append the entry to an already existing stream (if any) - labels := entry.Labels.String() + labels := labelsMapToString(entry.Labels, ReservedLabelTenantID) if stream, ok := b.streams[labels]; ok { stream.Entries = append(stream.Entries, entry.Entry) return @@ -53,6 +57,22 @@ func (b *batch) add(entry api.Entry) { } } +func labelsMapToString(ls model.LabelSet, without ...model.LabelName) string { + lstrs := make([]string, 0, len(ls)) +Outer: + for l, v := range ls { + for _, w := range without { + if l == w { + continue Outer + } + } + lstrs = append(lstrs, fmt.Sprintf("%s=%q", l, v)) + } + + sort.Strings(lstrs) + return fmt.Sprintf("{%s}", strings.Join(lstrs, ", ")) +} + // sizeBytes returns the current batch size in bytes func (b *batch) sizeBytes() int { return b.bytes diff --git a/pkg/promtail/client/client.go b/pkg/promtail/client/client.go index 2dbe134f1d8c..a57cf8251fdd 100644 --- a/pkg/promtail/client/client.go +++ b/pkg/promtail/client/client.go @@ -405,7 +405,6 @@ func (c *client) processEntry(e api.Entry) (api.Entry, string) { e.Labels = c.externalLabels.Merge(e.Labels) } tenantID := c.getTenantID(e.Labels) - delete(e.Labels, ReservedLabelTenantID) return e, tenantID } diff --git a/pkg/promtail/client/multi_test.go b/pkg/promtail/client/multi_test.go index 9d4e0a82d0a0..0b35c9ac74df 100644 --- a/pkg/promtail/client/multi_test.go +++ b/pkg/promtail/client/multi_test.go @@ -6,10 +6,13 @@ import ( "testing" "time" + "github.com/cortexproject/cortex/pkg/util" "github.com/cortexproject/cortex/pkg/util/flagext" util_log "github.com/cortexproject/cortex/pkg/util/log" + "github.com/go-kit/kit/log" "github.com/prometheus/client_golang/prometheus" "github.com/prometheus/common/model" + "github.com/stretchr/testify/require" "github.com/grafana/loki/pkg/logproto" "github.com/grafana/loki/pkg/promtail/api" @@ -115,3 +118,22 @@ func TestMultiClient_Handle(t *testing.T) { t.Fatal("missing handle call") } } + +func TestMultiClient_Handle_Race(t *testing.T) { + u := flagext.URLValue{} + require.NoError(t, u.Set("http://localhost")) + c1, err := New(nil, Config{URL: u, BackoffConfig: util.BackoffConfig{MaxRetries: 1}, Timeout: time.Microsecond}, log.NewNopLogger()) + require.NoError(t, err) + c2, err := New(nil, Config{URL: u, BackoffConfig: util.BackoffConfig{MaxRetries: 1}, Timeout: time.Microsecond}, log.NewNopLogger()) + require.NoError(t, err) + clients := []Client{c1, c2} + m := &MultiClient{ + clients: clients, + entries: make(chan api.Entry), + } + m.start() + + m.Chan() <- api.Entry{Labels: model.LabelSet{"foo": "bar", ReservedLabelTenantID: "1"}, Entry: logproto.Entry{Line: "foo"}} + + m.Stop() +} From babea82ef558c4b4dc685fd4ca677de1f0a7ee8f Mon Sep 17 00:00:00 2001 From: Edward Welch Date: Mon, 5 Apr 2021 20:36:38 -0400 Subject: [PATCH 9/9] prep release 2.2.1 --- CHANGELOG.md | 13 +++++++++++++ docs/sources/installation/docker.md | 18 +++++++++--------- docs/sources/upgrading/_index.md | 4 ++++ 3 files changed, 26 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9b9b3c588bba..242721535ed2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,16 @@ +## 2.2.1 (2021/04/05) + +2.2.1 fixes several important bugs, it is recommended everyone running 2.2.0 upgrade to 2.2.1 + +2.2.1 also adds the `labelallow` pipeline stage in Promtail which lets an allowlist be created for what labels will be sent by Promtail to Loki. + +* [3468](https://github.com/grafana/loki/pull/3468) **adityacs**: Support labelallow stage in Promtail +* [3502](https://github.com/grafana/loki/pull/3502) **cyriltovena**: Fixes a bug where unpack would mutate log line. +* [3540](https://github.com/grafana/loki/pull/3540) **cyriltovena**: Support for single step metric query. +* [3550](https://github.com/grafana/loki/pull/3550) **cyriltovena**: Fixes a bug in MatrixStepper when sharding queries. +* [3566](https://github.com/grafana/loki/pull/3566) **cyriltovena**: Properly release the ticker in Loki client. +* [3573](https://github.com/grafana/loki/pull/3573) **cyriltovena**: Fixes a race when using specific tenant and multi-client. + ## 2.2.0 (2021/03/10) With over 200 PR's 2.2 includes significant features, performance improvements, and bug fixes! diff --git a/docs/sources/installation/docker.md b/docs/sources/installation/docker.md index 73b373161f5c..50819112d8ea 100644 --- a/docs/sources/installation/docker.md +++ b/docs/sources/installation/docker.md @@ -18,10 +18,10 @@ For production, we recommend installing with Tanka or Helm. Copy and paste the commands below into your command line. ```bash -wget https://raw.githubusercontent.com/grafana/loki/v2.2.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml -docker run -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:2.2.0 -config.file=/mnt/config/loki-config.yaml -wget https://raw.githubusercontent.com/grafana/loki/v2.2.0/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml -docker run -v $(pwd):/mnt/config -v /var/log:/var/log grafana/promtail:2.2.0 -config.file=/mnt/config/promtail-config.yaml +wget https://raw.githubusercontent.com/grafana/loki/v2.2.1/cmd/loki/loki-local-config.yaml -O loki-config.yaml +docker run -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:2.2.1 -config.file=/mnt/config/loki-config.yaml +wget https://raw.githubusercontent.com/grafana/loki/v2.2.1/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml +docker run -v $(pwd):/mnt/config -v /var/log:/var/log grafana/promtail:2.2.1 -config.file=/mnt/config/promtail-config.yaml ``` When finished, `loki-config.yaml` and `promtail-config.yaml` are downloaded in the directory you chose. Docker containers are running Loki and Promtail using those config files. @@ -36,10 +36,10 @@ Copy and paste the commands below into your terminal. Note that you will need to ```bash cd "" -wget https://raw.githubusercontent.com/grafana/loki/v2.2.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml -docker run -v :/mnt/config -p 3100:3100 grafana/loki:2.2.0 --config.file=/mnt/config/loki-config.yaml -wget https://raw.githubusercontent.com/grafana/loki/v2.2.0/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml -docker run -v :/mnt/config -v /var/log:/var/log grafana/promtail:2.2.0 --config.file=/mnt/config/promtail-config.yaml +wget https://raw.githubusercontent.com/grafana/loki/v2.2.1/cmd/loki/loki-local-config.yaml -O loki-config.yaml +docker run -v :/mnt/config -p 3100:3100 grafana/loki:2.2.1 --config.file=/mnt/config/loki-config.yaml +wget https://raw.githubusercontent.com/grafana/loki/v2.2.1/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml +docker run -v :/mnt/config -v /var/log:/var/log grafana/promtail:2.2.1 --config.file=/mnt/config/promtail-config.yaml ``` When finished, `loki-config.yaml` and `promtail-config.yaml` are downloaded in the directory you chose. Docker containers are running Loki and Promtail using those config files. @@ -51,6 +51,6 @@ Navigate to http://localhost:3100/metrics to view the output. Run the following commands in your command line. They work for Windows or Linux systems. ```bash -wget https://raw.githubusercontent.com/grafana/loki/v2.2.0/production/docker-compose.yaml -O docker-compose.yaml +wget https://raw.githubusercontent.com/grafana/loki/v2.2.1/production/docker-compose.yaml -O docker-compose.yaml docker-compose -f docker-compose.yaml up ``` diff --git a/docs/sources/upgrading/_index.md b/docs/sources/upgrading/_index.md index e981ecb189a5..11c34cc14c09 100644 --- a/docs/sources/upgrading/_index.md +++ b/docs/sources/upgrading/_index.md @@ -19,6 +19,10 @@ If possible try to stay current and do sequential updates. If you want to skip v -_add changes here which are unreleased_ +## 2.2.1 + +Review the notes for 2.2.0, there are no additional update notes. + ## 2.2.0 ### Loki