pure plugin support file-to-file test #869
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
messixukejia
approved these changes
May 24, 2023
messixukejia
approved these changes
May 24, 2023
yyuuttaaoo
approved these changes
May 26, 2023
henryzhx8
pushed a commit
that referenced
this pull request
Jan 20, 2025
Link: https://code.alibaba-inc.com/sls/ilogtail/codereview/13013715 * Update ilogtail.md * feat: Define the metrics data model and modify the plugin manager to support procress the metrics data (#519) * Define the metrics data model and modify the plugin manager to support the new model * metric_mock,processor_default,flusher_stdout support metrics models * Optimize the metric print of stdout flusher * fix: Incorrect dnsPolicy setting in ilogtail-daemonset.yaml (#551) * fix: For pods using hostNetwork, their dnsPolicy should be set to "ClusterFirstWithHostNet" * update: The use case of kafka flusher changed to kafka flusher v2 * update: Brokers describe error * update: Remove wrong comma in example template * fix windows compatability (#556) * fix windows compatability * feat: add http flusher supporting influxdb protocol & add group aggregator supproting group logs by keys (#521) * feat: add influxdb flusher & group aggregator * feat: http flusher support concurrency setting * feat: do not replace influxdb acceptable chars when FieldsExtend enabled (#565) * feat: do not replace influxdb acceptable chars when FieldsExtend enabled * fix: add unit tests * fix: gofmt * fix 3 bugs in v1.3.0 (#574) * fix ptr bug (#566) * update processor_desensitize (#568) * fix arm version compatibility (#569) * fix buffer_file_path error (#573) * update version to 1.3.1 Co-authored-by: henryzhx8 <38356684+henryzhx8@users.noreply.github.com> Co-authored-by: Takuka0311 <66948919+Takuka0311@users.noreply.github.com> Co-authored-by: linrunqi08 <90741255+linrunqi08@users.noreply.github.com> * fix: fix metric name field name split error when contains ":" (#578) * fix: fix metric name field name split error when contains ":" * fix: fix unit tests * Define span model and byteArray model (#571) * Define the metrics data model and modify the plugin manager to support the new model * Fix build & lint error * metric_mock,processor_default,flusher_stdout support metrics models * update vscode ignore rule * update vscode ignore rule * Optimize the metric print of stdout flusher * Make SlsInput and PipelineInput compatible * fix InputMock * reset winmeta Opcodes * reset LocalCollector * reset codes * make both input run * reser * Fix metric_docker_file rdb mysql plugin run error * reset skywalking bp.go * reset skywalking pb.go * rename sls function name * reset * reset * reset * reset self_metric_imp_test * reset * reset * reset aggregators * reset aggregators * reset aggregators * refactor plugin runner * fix input mock * refactor: plugin runner * add pkg/constraints IntUintFloat * Fix restart pipeline * Fix flusher stop * Fix always run config tests * Add some ut * Add ut for plugin runner * Add math_helper_test * Add go version log * Update makefile * for test * reset constraints.IntUintFloat * refactor: rename plugin1 to xxxV1 % refactor pluginrunner * fix: Correctly handle the case comparison of version * refactor: Add Initialized check for pluginRunner * fix: support FlushOutStore overwrite * refactor: Optimize unsend data merge timing * define span and byteArray data model * resolve conflict * rename * rename * fix build error * add some notes * Fix gofmt lint * rename keyValuesEmpty to keyValuesNoop * Update metric-meta-host.md (#591) * Update system-config.md (#588) * Update build.md (#587) * Update performance-compare-with-filebeat.md (#590) * fix: decode error when input influxdb data contains multi fields (#583) * fix: decode error when input influxdb data contains multi fields * Update processor-desensitize.md (#593) fix some wrong descriptions * fix: metric labels compare error (#594) * fix time parsing problem (#552) * fix time parsing problem * feat: flusher kafka v2 support TLS and Kerberos (#602) * feat: flusher kafka support tls and Kerberos * change the order of dumping checkpoint to avoid checkpoint overwritten caused by inode multiplexing (#596) * change the order of dumping checkpoint * add container info (#567) add container info * fix prometheus stream fetch mode block (#607) Co-authored-by: liujiapeng.ljp <liujiapeng.ljp@alibaba-inc.com> * feat: add metadata group aggregator supporting group pipelineGroupEvents by metadata keys & enable http flusher and http server input supporting raw protocol (#599) * fix: KeyValuesImpl fix Contains func * fix: observePipeCollector fix ToArray func * feat: http server input support metric data model and raw format * feat: add metadata group aggregator * feat: http flusher support raw protocol * feat: stdout flusher support the events of byteArray type * feat: add docs for flusher/http-server, aggregator/metadata-group, input/http-service * fix: handle errors in writeByteArray for stdout flusher * fix: add DecoderV2 to interface Decoder * improve logs on the lifecycle of log reader and checkpoint (#595) * improve logs on the lifecycle of log reader and checkpoint * feat: http flusher add Convert.Separator config field (#616) * feat: http flusher add Convert.Separator config field * fix: add docs for http flusher convert separator * feat: add docs for raw protocol * feat: flusher pulsar (#603) * feat: flusher pulsar * fix blocking problem caused by alwaysonline config update (#622) * resume config before stopping it * refactor: move pipeline models to pkg module (#624) * refactor: move pipeline models to pkg module * Update ilogtail.md (#626) * add use-cases (#627) * Update developer-guide.md (#628) * 符号错误修正 (#629) * chatgroup update (#633) * update contributing (#635) * update contributing --------- Co-authored-by: Tao Yu <yyuuttaaoo@gmail.com> * fix: influxdb should allow empty string value (#631) * feat: otlp grpc/http input service for logs, metrics, and traces (v2) (#611) * feat: add oltp grpc input * feat: add cn docs for service_oltp_grpc * opentelemetry plugin supports http protocal * Grok processor report match errors by default (#645) * close #644, grok report match errors on default meanwhile change pure plugin LogtailSysConfDir to current dir fix plugin version in logs by using ldflag to set go plugin version * fix processor_desensitize (#640) * fix Chinese desensitize * feat: add clickhouse flusher (#554) (#580) * feat: add clickhouse flusher (#554) * test: complete the e2e test #554 * docs: docs update #554 * feat: automatically create buffer tables and support parameter configuration Signed-off-by: kl7sn <mex7.0828@gmail.com> Co-authored-by: Tao Yu <yyuuttaaoo@gmail.com> * feat: http flusher support convert Metric Event to influxdb protocol (#641) * feat: http flusher support convert Metric Event to influxdb protocol * feat: support config behavior on unexpected data * feat: http flusher support variable config in Header (#643) * feat: http flusher support variable config in Header * fix: lint * doc: http flusher header support variables * doc: add docs * feat: config server (#613) * Modified from pr#506 * Update AppConfig.cpp --------- Co-authored-by: henryzhx8 <38356684+henryzhx8@users.noreply.github.com> Co-authored-by: Tao Yu <yyuuttaaoo@gmail.com> * Fix metadata aggregator (#654) * fix-metadata-aggregator * fix-metadata-aggregator * fix-metadata-aggregator * fix-metadata-aggregator * fix-metadata-aggregator * polish codes --------- Co-authored-by: liujiapeng.ljp <liujiapeng.ljp@alibaba-inc.com> * feat: flusher_oltp plugin supports logs(v1) and metrics/traces(v2) (#646) * move otlp constant from helper to pkg to avoid import cycle * pass ut and link * flusher_otlp support seperate configs for logs/metrics/traces * update documents * feat: disable some output * fix ut * Revert "fix ut" This reverts commit 77f0330. * Revert "feat: disable some output" This reverts commit f409a76. * change: specific configturaions for logs/traces/metrics * fix typos in docs and logs * fix comments * make lint * rename ProtocolOtlpAllV1 to ProtocolOtlpV1 * remove unused constant * fix: skip leveKey * fix #661, fix update time to make container removable (#662) * fix #661, fix update time to make container removable * Fix docker user privilege problem (#664) * doc: configserver (#658) * Add doc on how to configure ConfigServer at Agent * Fix NPE Exception (#671) * 2022 open source report (#672) * feat: support private plugin integration (#638) * feat: support private plugin integration * fix: optimize code * fix: ignore git config -l error * fix: add import_plugins step for some Makefile command * fix: ignore git config -l error * fix: remove useless file * fix: modify Dockerfile according BUILD_KIT_ENABLE * fix: licenses * fix: unit tests * fix: e2e tests fix * doc: add changelog * doc: adjust plugin dev docs * doc: add plugins.yml config docs * fix: typo * fix: unit tests * Add pyroscope input new v1 (#673) * add pyroscope pprof profile support * polish codes * polish codes * polish codes * polish codes * polish codes * polish codes * add pyroscope pprof profile support * pyroscope tire support * pyroscope tire support * add pyroscope jfr (#604) * add jfr * supply go lables * supply go lables * polish codes * fix comment (#609) * polish codes * polish codes * polish codes * polish codes * polish codes * fix jfr data format (#610) * polish codes * polish codes * add profile input v2 * fix static check * fix static check * fix test * polish codes * add profile input v2 * add profile input v2 * add profile input v2 * add profile input v2 * polish codes * polish codes * polish codes * polish codes * polish codes * polish codes * polish codes * polish codes * polish codes * polish codes * polish codes * fix unit * fix copy error * polish codes * polish codes * polish codes * polish codes * polish codes * polish codes * polish codes * polish codes * polish codes * polish codes * polish codes * polish codes * polish codes * polish codes * remove v2 * polish codes * polish codes * polish codes * polish codes * polish codes * polish codes * polish codes * polish Pyroscope demo * polish Pyroscope demo * add changelog --------- Co-authored-by: liujiapeng.ljp <liujiapeng.ljp@alibaba-inc.com> Co-authored-by: linrunqi08 <90741255+linrunqi08@users.noreply.github.com> Co-authored-by: linrunqi08 <linrunqi08@163.com> * fix: ensure generating plugins-imports outside docker container (#682) * enhance prometheus support staleness (#684) * enhance prometheus * enhance prometheus * enhance prometheus * enhance prometheus --------- Co-authored-by: liujiapeng.ljp <liujiapeng.ljp@alibaba-inc.com> * Add logs for container matching in stdout plugin (#689) * Add logs for container matching in stdout plugin Set user agent for sls client * Update README.md * add more env field (#688) * add more env field to support sls logstore options * add hot ttl to product api * unify-profile-cpu-unit (#678) * unify-profile-cpu-unit * fix jfr unit * fix cpu and wall --------- Co-authored-by: liujiapeng.ljp <liujiapeng.ljp@alibaba-inc.com> Co-authored-by: linrunqi08 <linrunqi08@163.com> * compatible to windows compile (#694) * feat: suppprt log event model (#647) * feat: suppprt log event model * add switch for collect containers (#699) * add_switch_for_collect_containers * Add ecs meta (#692) * add ecs meta * add ecs meta * add ecs meta * add ecs meta * add ecs meta * add ecs meta * add ecs meta * add ecs meta * polish codes * add ecs meta * add ecs meta * add ecs meta * add ecs meta * polish codes * add ecs meta * fix lint * polish codes * polish codes * polish codes * polish codes * polish codes * polish codes * polish codes * polish codes * polish codes * polish codes * polish codes * polish codes * polish codes * polish codes * polish codes * polish codes * add examples * add examples * add examples * add examples --------- Co-authored-by: liujiapeng.ljp <liujiapeng.ljp@alibaba-inc.com> * add ecs meta to status metric (#709) * add ecs meta to status metric * fix comment * add ecs region id * try fix test * try fix test * reset test code * change property namespace to service.namespace (#718) * feat: retry on request timeout (#712) * add stability level doc (#705) * add stability level doc * add stability info for exsiting plugins * fix some bugs of configserver (#703) * fix some bugs * change int to Json::Value::ArrayIndex * Feat: Support elasticsearch flusher (#681) * Add flusher_elasticsearch.go * Add e2e test * Add licences * fix nil map panic (#721) * add test (#723) Co-authored-by: liujiapeng.ljp <liujiapeng.ljp@alibaba-inc.com> * Add async counter for unittest (#711) * add async counter helper * extend-http-dumper-ut-time * Update overview.md (#733) 修正错别字 * polish license (#736) Co-authored-by: liujiapeng.ljp <liujiapeng.ljp@alibaba-inc.com> * fix: fix a typo in .gitignore file. (#735) * bump version to 1.4.0 (#732) * bump version to 1.4.0 * polish license (#736) Co-authored-by: Evan <31562192+EvanLjp@users.noreply.github.com> * refine license (#739) Co-authored-by: liujiapeng.ljp <liujiapeng.ljp@alibaba-inc.com> * add fork license markdown (#741) Co-authored-by: liujiapeng.ljp <liujiapeng.ljp@alibaba-inc.com> * change github.com/mindprince/gonvml to github.com/iLogtail/gonvml (#722) * compatible to windows compile * change github.com/mindprince/gonvml to github.com/iLogtail/gonvml * license * merge form master * refine license * refine license * refine license * refine license * refine license * feat: add Extension type plugin && http flusher support Authenticator、Filter (#648) feat: add Extension type plugin && http flusher support Authenticator * fix broken link in SUMMARY.md (#748) * fix: unittest (#751) * fix: unittest * update CMakeLists * Add metrics system doc (#750) * add metric-system doc * temporarily remove nginx e2e case (#753) * improve doc * temporarily remove nginx e2e case * Fix: tag fields rename (#746) * fix #744 * update: open enable_env_ref_in_config * update: add addTagIfRequired function * update: optimised flusher-kafka_v2.md --------- Co-authored-by: Tom Yu <yyuuttaaoo@gmail.com> * feat: add loki flusher (#685) * feat: add loki flusher --------- Co-authored-by: henryzhx8 <38356684+henryzhx8@users.noreply.github.com> Co-authored-by: Tom Yu <yyuuttaaoo@gmail.com> * feat: Support elasticsearch dynamic index expression formatting (#717) * feat: Support elasticsearch dynamic index expression formatting * update: Add a document to introduce how to format the es index * fix: gozstd compatiblity in some environment (#761) * fix gozstd in some environment * feat: move helper to pkg/helper and implement decodeV2 of prometheus (#742) * prometheus decoder supports DecodeV2 * fix: missed import_plugins.sh (#763) * Fix ConfigServer's code (#743) * Standardize ConfigServer's code format. * Add mConfigServerAvailable * Jinchen #558 (#758) * upgrade: optimization processor_split_key_value plugin * feat: check docker version to determine whether using docker buildkit (#749) * feat: check docker version to determine whether using docker buildkit * feat: use docker buildkit only if docker_version >=19.03 && SSH_AUTH_SOCK env set * fix: set dockerfile syntax to enable --mount=type=ssh support on v19.03 * Processor regex replace (#757) * add processor_fields_with_condition to support process actions after switch-case conditions … (#139) * add processor_filter_compose_regex to support add fields after regex filter * [DOC] Add daemon tutorial (#769) * doc: add linux daemon * Update developer.md (#767) * Update developer.md * Update processor-regex.md (#772) add KeepSourceIfParseError in processor_regex doc * Feature add support of quote in processor_split_key_value plugin (#766) * feat: add support of quote in processor_split_key_value plugin * feat: add support of quote in processor_split_key_value plugin add tests * feat: add support of quote in processor_split_key_value add docs * Feature support goprofile pull (#764) * feature-support-goprofile-pull * support go profile pull * feature-support-goprofile-pull * feature-support-goprofile-pull * feature-support-goprofile-pull * feature-support-goprofile-pull * polish codes --------- Co-authored-by: liujiapeng.ljp <liujiapeng.ljp@alibaba-inc.com> * feat: otlp-log-v2 (#778) * otlp decoder implements decodev2 for logs * implement otlp log converter * fix: docs typo error (#783) * doc: fix example error of ext_groupinfo_filter (#785) * feat: move Log Body to LogContent (#800) * refactor Log Body too Contents * Fix test build error * update docs * [FIX] remove upperbound of autosplit (#797) * close #798 (#799) Fix index when creating logstore in query mode * feat: print response time in sender (#814) * add support for proxy through env (#790) * add support for proxy through env * add switch policy for sls data server endpoint (#707) * add switch policy for sls data server endpoint * ping add local mode support (#807) * fix: processor filter key filter log even if key matches (#815) * windows compile scripts (#745) * windows compile script --------- Co-authored-by: Tao Yu <yyuuttaaoo@gmail.com> * feat: support dynamic tag (#810) * feat: support dynamic tag * read dynamic tag from env ALIYUN_LOG_FILE_TAGS * bump version to 1.5.0 (#822) * add 1.5.0 artifacts * Update 1.5 docs (#823) * bump version to 1.5.0 (#822) * add 1.5.0 artifacts * fix windows 64-bit cgo interface * Merge branch '1.5' * Fix: stale env config (#825) * fix: do not read env config from exited containers * Merge branch '1.5' into main * Update SUMMARY.md (#833) * feat: optimize-flusher-pulsar (#819) * update: ptimize flusher pulsar to improve performance in static topic scenarios * update: format go.mod * update: add license * fix system disk total val (#835) * fix system disk total val --------- Co-authored-by: liujiapeng.ljp <liujiapeng.ljp@alibaba-inc.com> * Merge remote-tracking branch 'origin/1.5' * Documentation Improvement (#838) * Refactor and organize documentation * fix: processor_gotime empty time zone (#841) * fix env config cache (#844) * add tags to profile log (#840) * add tags to shennong log * fix compile * fix compile * fix comment * fix comment * Update start-with-k8s.md (#848) fix id / secret describe. * feat: implement influxdb DecodeV2 method (#846) * feat: implement influxdb DecodeV2 method * otlp decoder for metrics(v1) (#827) * otlp decoder for metrics(v1) * add exemplar * fix #851 (#852) * fix createAliyunLogOperationWrapper * fix fetch token interval * fix lint * fix create wrapper * fix UpdateTokenFunction * fix comment * fix comment * json converter marshal without HTML escaped (#857) * fix memory access error in evenv_handler (#858) * fix: should throw err on applyPluginConfig error (#856) * feat: upgrade kafka sarama client (#824) * feat: upgrade kafka sarama client to 1.38.1 * update: implements service input v2 * remove: remove MaxMessageLen option in service-kafka * skip empty tag values (#868) * feat: print imported plugin modules detail (#863) * pure plugin support file-to-file test (#869) * plugin_main support file to file * Add input line limit * close #872, fix container file fd not close if multiple configs monitor the same dir (#873) * fix bug (#876) * fix env config cache (#844) * add tags to profile log (#840) * add tags to shennong log * fix compile * fix compile * fix comment * fix comment * fix #851 (#852) * fix createAliyunLogOperationWrapper * fix fetch token interval * fix lint * fix create wrapper * fix UpdateTokenFunction * fix comment * fix comment * close #872, fix container file fd not close if multiple configs monitor the same dir (#873) * hotfix version 1.5.2 * Merge branch '1.5' into main * feat: prometheus decoder optimize (#866) * feat: pooling prometheus buf & request * fix: copybuffer bug fix * fix: reset timeseries * fix: zero copy prompb decode * fix: temp remove __hostname__ * feat: not recycle uncompressed request body * feat: simplify the snappy decode * feat: make prompb fields const * feat: add options for prometheus decoder unsafe mode * feat: restore the __hostname__ tag * fix: lint & license * fix: decoder options pass though * doc: note that exemplar not supported * doc: ref link for the prompb message definition * kubernetes add support kruise (#878) * kubernetes add support kruise * add license * add dependency license * add doc * feat: support set additional build flag via BUILD_FLAG env (#880) * feat: service-otlp-input supports gzip decompression (#874) * otlp-server supports gzip decompression * try to fix the gosec issue according to: securego/gosec#528 --------- Co-authored-by: Tom Yu <yyuuttaaoo@gmail.com> * add more prometheus sd support (#890) * Processor stringreplace fix bug (#892) * fix bug * add test * Processor desensitize improve performance (#891) * Improve performance * docs: explain processor_gotime time format (#901) Co-authored-by: Abingcbc <abingcbc626@163.com> * fix: gosec issue of tls version (#897) * fix gosec issue * fix shennong log key (#905) * fix static container (#906) * fix static container read logic * add test * user_contrib (#888) * user_contrib * add bash-history * doc: add config server ui quick start (#894) * doc: add config server ui quick start * fix: GET method replaced by POST to carry payload for browser * update CHANGELOG to explain the incompatible changes * fix multi line regex in cpp (#903) * fix multi-line log split * add ut * 1.6 pre release (#919) * bump version to 1.6.0 * update 1.6.0 release note * add window64 release script * change version for 1.6.0 * add plugins * merge open source 1.6.0 * rm unuse flushers * fix winuuid * rm vendor in gitignore * fix winuuid * fix program file path
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.