loki_out: add stuctured_metadata_map_keys #9530
Conversation
0x006EA1E5
requested review from
edsiper,
leonardo-albertovich,
fujimotos and
koleini
as code owners
0x006EA1E5
force-pushed
the
loki_out-structured_metadata_map
branch
from
7f1db76 to
a9cdcad
Compare
0x006EA1E5
force-pushed
the
loki_out-structured_metadata_map
branch
from
a9cdcad to
e9fee7d
Compare
* Adds stuctured_metadata_map_keys config to dynamically populate stuctured_metadata from a map Signed-off-by: Greg Eales <0x006EA1E5@gmail.com>
Signed-off-by: Greg Eales <0x006EA1E5@gmail.com>
0x006EA1E5
force-pushed
the
loki_out-structured_metadata_map
branch
from
dc24840 to
1f9301d
Compare
Signed-off-by: Greg Eales <0x006EA1E5@gmail.com>
| pack_label_key(mp_pck, (char*) accessed_map_kv.key.via.str.ptr, | ||
| accessed_map_kv.key.via.str.size); | ||
| /* | ||
| * Does this need optimising? For example, to handle |
There was a problem hiding this comment.
Can anyone answer this question? Should I optimise this somehow?
| @@ -78,6 +78,7 @@ struct flb_loki { | |||
| struct mk_list *labels; | |||
| struct mk_list *label_keys; | |||
| struct mk_list *structured_metadata; | |||
| struct mk_list *structured_metadata_map_keys; | |||
There was a problem hiding this comment.
is this config property name okay structured_metadata_map_keys?
| @@ -83,7 +83,7 @@ void flb_test_basic() | |||
| in_ffd = flb_input(ctx, (char *) "lib", NULL); | |||
| flb_input_set(ctx, in_ffd, "tag", "test", NULL); | |||
|
|
|||
| /* Elasticsearch output */ | |||
There was a problem hiding this comment.
I am assuming that all these should have said Loki and not Elasticsearch, and that it is okay to chnage them in this PR
| "{\"sub_key1\": \"sub_value1\", \"sub_key2\": false}}, \"map2\": {\"key4\": " \ | ||
| "\"value1\", \"key5\": false}, \"map3\": {\"key1\": \"map3_value1\", \"key2\": " \ | ||
| "\"map3_value2\"}}]" | ||
| void flb_test_structured_metadata_map_params(char *remove_keys, |
There was a problem hiding this comment.
There are quite a few different variations on this test, so I have created this function which takes various parameters, and then called it for each actual test (below)
| {"structured_metadata_map_and_explicit", | ||
| flb_test_structured_metadata_map_and_explicit}, | ||
| {"structured_metadata_map_single_missing_map", | ||
| flb_test_structured_metadata_map_single_missing_map}, |
There was a problem hiding this comment.
Any test cases missing that I should add?
| &ctx->structured_metadata_map_keys_list); | ||
| } | ||
| /* | ||
| * explicit structured_metadata entries override |
There was a problem hiding this comment.
I believe this works as expected.
The intention is, if we try to add an entry to the structured_metadata with the same key twice (first from the new structured_metadata_map_keys, then from the existing structured_metadata), then the second entry should "win" and overrwite the first.
I am assuming msgpack works this way. But maybe I actually need to do some kind of explicit check, and only add a new entry where there is not already one for a given key?
| * separate item. Non-string map values are serialised to JSON, as Loki requires | ||
| * all values to be strings. | ||
| */ | ||
| static void pack_maps(struct flb_loki *ctx, |
There was a problem hiding this comment.
This essentially tries to follow the flow of the proceeding pack_kv function.
Do you need a more detailed spec of what exactly this is supposed to be doing?
It should implement the proposal descibed in the FR #9463
|
I added a PR for the docs here fluent/fluent-bit-docs#1527 |
Addresses #9463
Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
ok-package-testlabel to test for all targets (requires maintainer to do).Documentation
fluent/fluent-bit-docs#1527
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.