Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

processor_metrics_selector: Implement delete by label value operation #8812

Merged
merged 2 commits into from
Jun 7, 2024
Merged

processor_metrics_selector: Implement delete by label value operation #8812

merged 2 commits into from
Jun 7, 2024

Conversation

cosmo0920
Copy link
Contributor

@cosmo0920 cosmo0920 commented May 10, 2024 •
edited
Loading

In this PR, I implemented a context of delete_label_value operation.

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
service:
  flush: 2
  daemon: off
  log_level:  debug

pipeline:
  inputs:
    - name: fluentbit_metrics
      tag: fluentbit.metrics
      scrape_interval: 2

      processors:
        metrics:
          - name: metrics_selector
            context: delete_label_value
            label: name stdout.0

  outputs:
    - name: stdout
      match: '*'
  • Debug log output from testing the change
Fluent Bit v3.0.4
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

___________.__                        __    __________.__  __          ________  
\_   _____/|  |  __ __   ____   _____/  |_  \______   \__|/  |_  ___  _\_____  \ 
 |    __)  |  | |  |  \_/ __ \ /    \   __\  |    |  _/  \   __\ \  \/ / _(__  < 
 |     \   |  |_|  |  /\  ___/|   |  \  |    |    |   \  ||  |    \   / /       \
 \___  /   |____/____/  \___  >___|  /__|    |______  /__||__|     \_/ /______  /
     \/                     \/     \/               \/                        \/ 

[2024/05/10 15:55:59] [ info] Configuration:
[2024/05/10 15:55:59] [ info]  flush time     | 2.000000 seconds
[2024/05/10 15:55:59] [ info]  grace          | 5 seconds
[2024/05/10 15:55:59] [ info]  daemon         | 0
[2024/05/10 15:55:59] [ info] ___________
[2024/05/10 15:55:59] [ info]  inputs:
[2024/05/10 15:55:59] [ info]      fluentbit_metrics
[2024/05/10 15:55:59] [ info] ___________
[2024/05/10 15:55:59] [ info]  filters:
[2024/05/10 15:55:59] [ info] ___________
[2024/05/10 15:55:59] [ info]  outputs:
[2024/05/10 15:55:59] [ info]      stdout.0
[2024/05/10 15:55:59] [ info] ___________
[2024/05/10 15:55:59] [ info]  collectors:
[2024/05/10 15:55:59] [ info] [fluent bit] version=3.0.4, commit=41ef155add, pid=138532
[2024/05/10 15:55:59] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2024/05/10 15:55:59] [ info] [storage] ver=1.1.6, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/05/10 15:55:59] [ info] [cmetrics] version=0.9.0
[2024/05/10 15:55:59] [ info] [ctraces ] version=0.5.1
[2024/05/10 15:55:59] [ info] [input:fluentbit_metrics:fluentbit_metrics.0] initializing
[2024/05/10 15:55:59] [ info] [input:fluentbit_metrics:fluentbit_metrics.0] storage_strategy='memory' (memory only)
[2024/05/10 15:55:59] [debug] [fluentbit_metrics:fluentbit_metrics.0] created event channels: read=21 write=22
[2024/05/10 15:55:59] [debug] [stdout:stdout.0] created event channels: read=23 write=24
[2024/05/10 15:55:59] [ info] [output:stdout:stdout.0] worker #0 started
[2024/05/10 15:55:59] [ info] [sp] stream processor started
[2024/05/10 15:56:03] [debug] [task] created task=0x5f52390 id=0 OK
[2024/05/10 15:56:03] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
2024-05-10T06:56:01.036525578Z fluentbit_uptime{hostname="cosmo-desktop2"} = 2
2024-05-10T06:55:59.414406238Z fluentbit_input_bytes_total{name="fluentbit_metrics.0"} = 0
2024-05-10T06:55:59.414406238Z fluentbit_input_records_total{name="fluentbit_metrics.0"} = 0
2024-05-10T06:56:01.032061668Z fluentbit_input_metrics_scrapes_total{name="fluentbit_metrics.0"} = 1
2024-05-10T06:56:01.036525578Z fluentbit_process_start_time_seconds{hostname="cosmo-desktop2"} = 1715324159
2024-05-10T06:56:01.036525578Z fluentbit_build_info{hostname="cosmo-desktop2",version="3.0.4",os="linux"} = 1715324159
2024-05-10T06:56:01.036525578Z fluentbit_hot_reloaded_times{hostname="cosmo-desktop2"} = 0
2024-05-10T06:56:01.045084677Z fluentbit_storage_chunks = 0
2024-05-10T06:56:01.045084677Z fluentbit_storage_mem_chunks = 0
2024-05-10T06:56:01.045084677Z fluentbit_storage_fs_chunks = 0
2024-05-10T06:56:01.045084677Z fluentbit_storage_fs_chunks_up = 0
2024-05-10T06:56:01.045084677Z fluentbit_storage_fs_chunks_down = 0
2024-05-10T06:55:59.414406238Z fluentbit_input_ingestion_paused{name="fluentbit_metrics.0"} = 0
2024-05-10T06:55:59.414406238Z fluentbit_input_storage_overlimit{name="fluentbit_metrics.0"} = 0
2024-05-10T06:55:59.414406238Z fluentbit_input_storage_memory_bytes{name="fluentbit_metrics.0"} = 0
2024-05-10T06:55:59.414406238Z fluentbit_input_storage_chunks{name="fluentbit_metrics.0"} = 0
2024-05-10T06:55:59.414406238Z fluentbit_input_storage_chunks_up{name="fluentbit_metrics.0"} = 0
2024-05-10T06:55:59.414406238Z fluentbit_input_storage_chunks_down{name="fluentbit_metrics.0"} = 0
2024-05-10T06:55:59.414406238Z fluentbit_input_storage_chunks_busy{name="fluentbit_metrics.0"} = 0
2024-05-10T06:55:59.414406238Z fluentbit_input_storage_chunks_busy_bytes{name="fluentbit_metrics.0"} = 0
[2024/05/10 15:56:03] [debug] [out flush] cb_destroy coro_id=0
[2024/05/10 15:56:03] [debug] [task] destroy task=0x5f52390 (task_id=0)
^C[2024/05/10 15:56:03] [engine] caught signal (SIGINT)
[2024/05/10 15:56:03] [debug] [task] created task=0x60040e0 id=0 OK
[2024/05/10 15:56:03] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
2024-05-10T06:56:03.049517846Z fluentbit_uptime{hostname="cosmo-desktop2"} = 4
2024-05-10T06:55:59.414406238Z fluentbit_input_bytes_total{name="fluentbit_metrics.0"} = 0
2024-05-10T06:55:59.414406238Z fluentbit_input_records_total{name="fluentbit_metrics.0"} = 0
2024-05-10T06:56:03.049340936Z fluentbit_input_metrics_scrapes_total{name="fluentbit_metrics.0"} = 2
2024-05-10T06:56:03.049517846Z fluentbit_process_start_time_seconds{hostname="cosmo-desktop2"} = 1715324159
2024-05-10T06:56:03.049517846Z fluentbit_build_info{hostname="cosmo-desktop2",version="3.0.4",os="linux"} = 1715324159
2024-05-10T06:56:03.049517846Z fluentbit_hot_reloaded_times{hostname="cosmo-desktop2"} = 0
2024-05-10T06:56:03.049862719Z fluentbit_storage_chunks = 1
2024-05-10T06:56:03.049862719Z fluentbit_storage_mem_chunks = 1
2024-05-10T06:56:03.049862719Z fluentbit_storage_fs_chunks = 0
2024-05-10T06:56:03.049862719Z fluentbit_storage_fs_chunks_up = 0
2024-05-10T06:56:03.049862719Z fluentbit_storage_fs_chunks_down = 0
2024-05-10T06:55:59.414406238Z fluentbit_input_ingestion_paused{name="fluentbit_metrics.0"} = 0
2024-05-10T06:55:59.414406238Z fluentbit_input_storage_overlimit{name="fluentbit_metrics.0"} = 0
2024-05-10T06:55:59.414406238Z fluentbit_input_storage_memory_bytes{name="fluentbit_metrics.0"} = 0
2024-05-10T06:55:59.414406238Z fluentbit_input_storage_chunks{name="fluentbit_metrics.0"} = 0
2024-05-10T06:55:59.414406238Z fluentbit_input_storage_chunks_up{name="fluentbit_metrics.0"} = 0
[2024/05/10 15:56:03] [ warn] [engine] service will shutdown in max 5 seconds
2024-05-10T06:55:59.414406238Z fluentbit_input_storage_chunks_down{name="fluentbit_metrics.0"} = 0
2024-05-10T06:55:59.414406238Z fluentbit_input_storage_chunks_busy{name="fluentbit_metrics.0"} = 0
2024-05-10T06:55:59.414406238Z fluentbit_input_storage_chunks_busy_bytes{name="fluentbit_metrics.0"} = 0
[2024/05/10 15:56:03] [debug] [out flush] cb_destroy coro_id=1
[2024/05/10 15:56:03] [ info] [input] pausing fluentbit_metrics.0
[2024/05/10 15:56:03] [debug] [task] destroy task=0x60040e0 (task_id=0)
[2024/05/10 15:56:04] [ info] [engine] service has stopped (0 pending tasks)
[2024/05/10 15:56:04] [ info] [input] pausing fluentbit_metrics.0
[2024/05/10 15:56:04] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2024/05/10 15:56:04] [ info] [output:stdout:stdout.0] thread worker #0 stopped
  • Attached Valgrind output that shows no leaks or memory corruption was found
==138532== 
==138532== HEAP SUMMARY:
==138532==     in use at exit: 0 bytes in 0 blocks
==138532==   total heap usage: 8,187 allocs, 8,187 frees, 1,616,398 bytes allocated
==138532== 
==138532== All heap blocks were freed -- no leaks are possible
==138532== 
==138532== For lists of detected and suppressed errors, rerun with: -s
==138532== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

fluent/fluent-bit-docs#1367

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@cosmo0920 cosmo0920 mentioned this pull request May 10, 2024
Merged
@edsiper
Copy link
Member

edsiper commented May 29, 2024

note: this is still a draft

@edsiper edsiper added this to the Fluent Bit v3.0.7 milestone May 29, 2024
@cosmo0920 cosmo0920 force-pushed the cosmo0920-implement-delete-by-label-value-operation-on-processor_metrics_selector branch from e73334c to 3967d7b Compare May 30, 2024 06:14
@cosmo0920
Copy link
Contributor Author

I've resolved a conflict. This PR status is for waiting a new version of cmetrics which includes a deleting metrics feature by label's key-value pair.

@cosmo0920
processor_metrics_selector: Implement to delete by label key-value pa…
d74986b 
…ir feature

Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
@cosmo0920
processor_metrics_selector: Add test case for delete_label_value oper…
d6a0712 
…ation

Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
@cosmo0920 cosmo0920 force-pushed the cosmo0920-implement-delete-by-label-value-operation-on-processor_metrics_selector branch from 3967d7b to d6a0712 Compare June 6, 2024 14:01
@cosmo0920 cosmo0920 marked this pull request as ready for review June 6, 2024 14:01
@cosmo0920
Copy link
Contributor Author

Done for rebasing off the current master.

@edsiper edsiper merged commit f27a884 into master Jun 7, 2024
46 checks passed
@edsiper edsiper deleted the cosmo0920-implement-delete-by-label-value-operation-on-processor_metrics_selector branch June 7, 2024 02:15
@BrewTestBot BrewTestBot mentioned this pull request Jun 12, 2024
Merged
@BrewTestBot BrewTestBot mentioned this pull request Jul 9, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@edsiper edsiper Awaiting requested review from edsiper edsiper is a code owner

@leonardo-albertovich leonardo-albertovich Awaiting requested review from leonardo-albertovich leonardo-albertovich is a code owner

@fujimotos fujimotos Awaiting requested review from fujimotos fujimotos is a code owner

@koleini koleini Awaiting requested review from koleini koleini is a code owner

Assignees
No one assigned
Labels
docs-required
Projects
None yet
Milestone
Fluent Bit v3.0.7
Development

Successfully merging this pull request may close these issues.
2 participants
@cosmo0920 @edsiper