out_loki: support new option tenant_id_key(#2935) #3260

nokute78 · 2021-03-20T00:50:17Z

Fixes #2935

This patch support new option tenant_id_key.
The option allows record accessor for nested value.

Note:
If different values are sent to a loki plugin at once, X-Scope-OrdID will be set the value of last record.

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

Example configuration file for the change
Debug log output from testing the change

Attached Valgrind output that shows no leaks or memory corruption was found

Documentation

Documentation required for this feature

Example Cofiguration

Fluent-bit will send a dummy record which has "tenant_id":"MyTenant".
This configuration is to set tenant_id_key and fluent-bit will set MyTenant as X-Scope-OrgID.

a.conf:

[INPUT]
    name dummy
    dummy    {"tenant_id":"MyTenant", "data":"hoge", "nested":{"key":"MyTenant2"}}

[OUTPUT]
    name loki
    tenant_id_key tenant_id

## also support nested key
#    tenant_id_key $nested['key']

Debug log

This testing log using nc to check X-Scope-OrgID.

exec nc -l 3100
exec fluent-bit to send nc

nc -l 3100
$ ../bin/fluent-bit -c a.conf 
Fluent Bit v1.8.0
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2021/03/20 09:41:19] [ info] [engine] started (pid=24809)
[2021/03/20 09:41:19] [ info] [storage] version=1.1.1, initializing...
[2021/03/20 09:41:19] [ info] [storage] in-memory
[2021/03/20 09:41:19] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/03/20 09:41:19] [ info] [output:loki:loki.0] configured, hostname=127.0.0.1:3100
[2021/03/20 09:41:19] [ info] [sp] stream processor started
^C[2021/03/20 09:41:27] [engine] caught signal (SIGINT)
[2021/03/20 09:41:27] [ warn] [engine] service will stop in 5 seconds
[2021/03/20 09:41:32] [ info] [engine] service stopped
[2021/03/20 09:41:32] [ warn] [engine] shutdown delayed, grace period has finished but some tasks are still running.
[2021/03/20 09:41:32] [ info] [task] dummy/dummy.0 has 2 pending task(s):
[2021/03/20 09:41:32] [ info] [task]   task_id=0 still running on route(s): loki/loki.0 
[2021/03/20 09:41:32] [ info] [task]   task_id=1 still running on route(s): loki/loki.0 
[2021/03/20 09:41:32] [ warn] [engine] service will stop in 5 seconds
^C[2021/03/20 09:41:33] [engine] caught signal (SIGINT)

nc ouput is here. The point is X-Scope-OrgID: MyTenant

$ nc -l 3100
POST /loki/api/v1/push HTTP/1.1
Host: 127.0.0.1:3100
Content-Length: 488
User-Agent: Fluent-Bit
Content-Type: application/json
X-Scope-OrgID: MyTenant

{"streams":[{"stream":{"job":"fluent-bit"},"values":[["1616200879944358396","{\"tenant_id\":\"MyTenant\",\"data\":\"hoge\",\"nested\":{\"key\":\"MyTenant2\"}}"],["1616200880944288024","{\"tenant_id\":\"MyTenant\",\"data\":\"hoge\",\"nested\":{\"key\":\"MyTenant2\"}}"],["1616200881944318795","{\"tenant_id\":\"MyTenant\",\"data\":\"hoge\",\"nested\":{\"key\":\"MyTenant2\"}}"],["1616200882944294999","{\"tenant_id\":\"MyTenant\",\"data\":\"hoge\",\"nested\":{\"key\":\"MyTenant2\"}}"]]}]}

Valgrind output

Launch loki using docker before testing.

wget https://raw.githubusercontent.com/grafana/loki/v2.2.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml
docker run -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:2.2.0 -config.file=/mnt/config/loki-config.yaml

$ valgrind --leak-check=full ../bin/fluent-bit -c a.conf 
==24737== Memcheck, a memory error detector
==24737== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==24737== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==24737== Command: ../bin/fluent-bit -c a.conf
==24737== 
Fluent Bit v1.8.0
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2021/03/20 09:39:29] [ info] [engine] started (pid=24737)
[2021/03/20 09:39:29] [ info] [storage] version=1.1.1, initializing...
[2021/03/20 09:39:29] [ info] [storage] in-memory
[2021/03/20 09:39:29] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/03/20 09:39:29] [ info] [output:loki:loki.0] configured, hostname=127.0.0.1:3100
[2021/03/20 09:39:29] [ info] [sp] stream processor started
==24737== Warning: client switching stacks?  SP change: 0x57e48b8 --> 0x4c76750
==24737==          to suppress, use: --max-stackframe=11985256 or greater
==24737== Warning: client switching stacks?  SP change: 0x4c766c8 --> 0x57e48b8
==24737==          to suppress, use: --max-stackframe=11985392 or greater
==24737== Warning: client switching stacks?  SP change: 0x57e48b8 --> 0x4c766c8
==24737==          to suppress, use: --max-stackframe=11985392 or greater
==24737==          further instances of this message will not be shown.
[2021/03/20 09:39:34] [ info] [output:loki:loki.0] 127.0.0.1:3100, HTTP status=204
^C[2021/03/20 09:39:35] [engine] caught signal (SIGINT)
[2021/03/20 09:39:35] [ info] [output:loki:loki.0] 127.0.0.1:3100, HTTP status=204
[2021/03/20 09:39:35] [ warn] [engine] service will stop in 5 seconds
[2021/03/20 09:39:39] [ info] [engine] service stopped
==24737== 
==24737== HEAP SUMMARY:
==24737==     in use at exit: 0 bytes in 0 blocks
==24737==   total heap usage: 468 allocs, 468 frees, 1,211,941 bytes allocated
==24737== 
==24737== All heap blocks were freed -- no leaks are possible
==24737== 
==24737== For lists of detected and suppressed errors, rerun with: -s
==24737== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Signed-off-by: Takahiro Yamashita <nokute78@gmail.com>

nokute78 · 2021-03-20T08:17:20Z

This patch is draft.
Because the patch is not for real kubernetes usecase.
#2935 (comment)

nokute78 · 2021-03-27T23:51:27Z

Remove draft status.

Because rewrite_tag and this patch can solve #2935

nokute78 · 2021-03-30T03:40:52Z

The PR is ready to review.

This is an example of the new property.

Document example

key	description	default
tenant_id_key	Specify the name of the key from the original record that contains the Tenant ID. It is useful to set Tenant ID dynamically.

edsiper · 2021-03-30T04:12:12Z

please check the unit tests status

nokute78 · 2021-03-30T04:28:13Z

@edsiper I re-run CI.

There two error cases.

commit message
- It is fixed by workflow: allow reference message #3285
Setup environment failure (almost all cases)
- I think it is not related this PR.

Err:1 http://azure.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libsystemd-dev amd64 237-3ubuntu10.44
  404  Not Found [IP: 52.250.76.244 80]
Get:2 http://azure.archive.ubuntu.com/ubuntu bionic/universe amd64 gcovr all 3.4-1 [24.7 kB]
E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/main/s/systemd/libsystemd-dev_237-3ubuntu10.44_amd64.deb  404  Not Found [IP: 52.250.76.244 80]
Fetched 24.7 kB in 0s (268 kB/s)
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
Error: Process completed with exit code 100.

nokute78 · 2021-03-30T04:37:41Z

@edsiper Oops commit message regex is not updated by re-run...
It is old pattern which means #3285 is not merged in this PR...

I tested using current pattern by hand and it reports ok.

$ irb 
irb(main):001:0> str = 'out_loki: support new option tenant_id_key(#2935)'
irb(main):002:0> str.match /^[a-z\-]+\:[ ]{0,1}[a-z]+[a-zA-Z0-9 \-\.\:]+$/
=> nil
irb(main):003:0> str.match /^[a-z0-9\-_]+\:[ ]{0,1}[a-z]+[a-zA-Z0-9 \-\.\:_\#\(\)]+$/
=> #<MatchData "out_loki: support new option tenant_id_key(#2935)">

002:0 is old pattern which CI of this PR uses.
003:0 is current pattern.

nokute78 · 2021-03-30T07:31:16Z

I re-run CI and only Commit Message Check is failed.
It is because the commit message regex is old.

I tested it by hand using current regex and it reports ok.
#3260 (comment)

So, CI is all OK and we can merge it.

nokute78 · 2021-04-12T11:06:10Z

@edsiper Ping.

CI error should be recovered current master regex pattern.
See also #3260 (comment)

edsiper · 2021-05-05T21:51:23Z

thanks

out_loki: support new option tenant_id_key(fluent#2935)

c47a81b

Signed-off-by: Takahiro Yamashita <nokute78@gmail.com>

nokute78 requested review from edsiper, fujimotos and koleini as code owners March 20, 2021 00:50

nokute78 mentioned this pull request Mar 20, 2021

[Loki Plugin] Allow setting X-Scope-OrgID header dynamically #2935

Closed

nokute78 marked this pull request as draft March 20, 2021 08:15

nokute78 marked this pull request as ready for review March 27, 2021 23:50

edsiper merged commit 592d401 into fluent:master May 5, 2021

nokute78 mentioned this pull request May 6, 2021

out_loki: add description about 'tenant_id_key' fluent/fluent-bit-docs#529

Merged

nokute78 deleted the fix_2935_loki_dynamic_id branch May 6, 2021 11:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

out_loki: support new option tenant_id_key(#2935) #3260

out_loki: support new option tenant_id_key(#2935) #3260

nokute78 commented Mar 20, 2021

nokute78 commented Mar 20, 2021

nokute78 commented Mar 27, 2021

nokute78 commented Mar 30, 2021

edsiper commented Mar 30, 2021

nokute78 commented Mar 30, 2021

nokute78 commented Mar 30, 2021

nokute78 commented Mar 30, 2021

nokute78 commented Apr 12, 2021

edsiper commented May 5, 2021

out_loki: support new option tenant_id_key(#2935) #3260

out_loki: support new option tenant_id_key(#2935) #3260

Conversation

nokute78 commented Mar 20, 2021

Example Cofiguration

Debug log

Valgrind output

nokute78 commented Mar 20, 2021

nokute78 commented Mar 27, 2021

nokute78 commented Mar 30, 2021

Document example

edsiper commented Mar 30, 2021

nokute78 commented Mar 30, 2021

nokute78 commented Mar 30, 2021

nokute78 commented Mar 30, 2021

nokute78 commented Apr 12, 2021

edsiper commented May 5, 2021