feat: support continue/end multiple regex patterns #978

Abingcbc · 2023-07-05T07:47:32Z

配置文件

enable: true
inputs:
  - Type: file_log
    LogPath: .
    FilePattern: simple.log
processors:
  - Type: processor_regex_accelerate
    Keys:
    - msg
    - time
    Regex: (\[(\S+)].*)
    DiscardUnmatch: false
    LogBeginRegex: \[\d+-\d+-\w+:\d+:\d+.\d+].*Exception.*
    LogEndRegex: .*\.\.\..*
flushers:
  - Type: flusher_sls
    Endpoint: 
    ProjectName:
    LogstoreName:

样例日志

[2022-07-07T10:43:27.360266763] [ERROR] java.lang.Exception: exception happened
[2022-07-07T10:43:27.360266763]    at com.aliyun.sls.devops.logGenerator.type.RegexMultiLog.f2(RegexMultiLog.java:108)
[2022-07-07T10:43:27.360266763]    at java.base/java.lang.Thread.run(Thread.java:833)
[2022-07-07T10:43:27.360266763]    ... 23 more
[2022-07-07T10:43:27.360266763] Some user custom log
[2022-07-07T10:43:27.360266763] Some user custom log
[2022-07-07T10:43:27.360266763] Some user custom log
[2022-07-07T10:43:27.360266763] Some user custom log
[2022-07-07T10:43:27.360266763] [ERROR] java.lang.Exception: exception happened
[2022-07-07T10:43:27.360266763]    at com.aliyun.sls.devops.logGenerator.type.RegexMultiLog.f2(RegexMultiLog.java:108)
[2022-07-07T10:43:27.360266763]    at java.base/java.lang.Thread.run(Thread.java:833)
[2022-07-07T10:43:27.360266763]    ... 23 more
[2022-07-07T10:43:27.360266763] Some user custom log
[2022-07-07T10:43:27.360266763] Some user custom log

结果

多行

单行

core/config/Config.cpp

core/config_manager/ConfigYamlToJson.cpp

core/reader/LogFileReader.h

core/config/Config.cpp

docs/cn/data-pipeline/accelerator/regex-accelerate.md

core/reader/LogFileReader.h

core/reader/LogFileReader.cpp

docs/cn/data-pipeline/accelerator/regex-accelerate.md

core/reader/LogFileReader.cpp

core/reader/LogFileReader.h

core/unittest/config/ConfigMatchUnittest.cpp

core/reader/LogFileReader.cpp

core/unittest/reader/LastMatchedLineUnittest.cpp

core/unittest/reader/LogSplitUnittest.cpp

docs/cn/data-pipeline/accelerator/regex-accelerate.md

* close #886, refactory LogBuffer structure refine memory management support \0 in log * add unittest * close #886, refactory LogBuffer structure refine memory management support \0 in log * add unittest * close #886, refactory LogBuffer structure (#930) * close #886, refactory LogBuffer structure refine memory management support \0 in log * add unittest * close #886, refactory LogBuffer structure refine memory management support \0 in log * add unittest * test: complement test cases for reader and parser (#946) * fix unittest timezone * reader unit test * fix: multi-bytes character cut off (#968) * fix: multi-bytes character cut off * support GB 18030 * [WIP] C++ pipeline stage1: processors new model and partial processors without unittests * do not commit personalized settings * remove unused fields in data structure * fix compile error * C++ pipeline stage1: processors (#995) * remove useless codes * add UT * Remove the limit that accelerate plugins cannot be used with other plugins * fix accelerate plugin with only go flusher plugin * fix UT * fix member uninitialized * fix pass to Go plugin tag name error * fix c++ to go process * fix process buffer legacy * refine code according to comments * stick to container naming convention, stick to smart pointer param passing convention * fix split line if not enable_new_pipeline * accelerate FileSystemUtilUnittest speed Signed-off-by: Tao Yu <yyuuttaaoo@gmail.com> * fix multiline config when using new pipeline (#1079) * fix multiline config when using new pipeline * refine doc * fix pure plugin multiline does not split correctly --------- Signed-off-by: Tao Yu <yyuuttaaoo@gmail.com> * feat: support continue/end multiple regex patterns (#978) * feat: support continue/end multiple regex patterns * fix unittest * doc --------- Co-authored-by: Tom Yu <yyuuttaaoo@gmail.com> * feat: read timeout (#1067) * feat: read log timeout * fix ci with longer timeout * remove profiler in CMakeLists.txt Signed-off-by: yyuuttaaoo <yyuuttaaoo@gmail.com> * add changelog --------- Signed-off-by: Tao Yu <yyuuttaaoo@gmail.com> Signed-off-by: yyuuttaaoo <yyuuttaaoo@gmail.com> Co-authored-by: Bingchang Chen <abingcbc626@gmail.com>

yyuuttaaoo requested review from yyuuttaaoo and messixukejia July 5, 2023 09:14

Abingcbc force-pushed the improve-multiline branch from bb041fa to 3d0a4d6 Compare July 6, 2023 08:23

messixukejia requested changes Jul 9, 2023

View reviewed changes

core/config/Config.cpp Outdated Show resolved Hide resolved

core/config_manager/ConfigYamlToJson.cpp Show resolved Hide resolved

core/reader/LogFileReader.h Outdated Show resolved Hide resolved

Abingcbc marked this pull request as draft July 10, 2023 01:57

Abingcbc changed the title ~~feat: support multiple regex pattern in LastMatchedLine~~ WIP: feat: support multiple regex patterns Jul 10, 2023

Abingcbc force-pushed the improve-multiline branch 3 times, most recently from 45ad3e8 to ad46fb4 Compare July 12, 2023 08:59

Abingcbc marked this pull request as ready for review July 12, 2023 08:59

Abingcbc changed the title ~~WIP: feat: support multiple regex patterns~~ feat: support multiple regex patterns Jul 12, 2023

Abingcbc force-pushed the improve-multiline branch from 36d5474 to 62e525c Compare July 14, 2023 09:31

yyuuttaaoo reviewed Jul 16, 2023

View reviewed changes

messixukejia requested changes Jul 18, 2023

View reviewed changes

yyuuttaaoo requested changes Aug 5, 2023

View reviewed changes

Abingcbc force-pushed the improve-multiline branch 2 times, most recently from 94ff7b0 to 987314f Compare August 10, 2023 08:26

yyuuttaaoo reviewed Aug 16, 2023

View reviewed changes

core/unittest/reader/LogSplitUnittest.cpp Outdated Show resolved Hide resolved

core/unittest/reader/LogSplitUnittest.cpp Show resolved Hide resolved

Abingcbc force-pushed the improve-multiline branch from 4fe5543 to 7ddb047 Compare August 17, 2023 01:54

Abingcbc added 9 commits August 17, 2023 10:01

feat: support multiple regex patterns

f623653

fix unittest

52001b3

doc

659c51c

optimize code

5cb5c8b

simplify the logic

bdef3a6

optimize code

a3e2f13

fix comments

ca6a973

fix conflict

2a5168a

fix rebase

610bcba

Abingcbc force-pushed the improve-multiline branch from 7ddb047 to 610bcba Compare August 17, 2023 02:05

fix

9c05b74

fix

56be07c

yyuuttaaoo approved these changes Aug 20, 2023

View reviewed changes

messixukejia approved these changes Aug 21, 2023

View reviewed changes

docs/cn/data-pipeline/accelerator/regex-accelerate.md Show resolved Hide resolved

fix doc

3385997

Takuka0311 approved these changes Aug 21, 2023

View reviewed changes

Merge branch 'feature/memory_model' into improve-multiline

50b3993

yyuuttaaoo changed the title ~~feat: support multiple regex patterns~~ feat: support continue/end multiple regex patterns Aug 24, 2023

yyuuttaaoo merged commit e6f0be8 into alibaba:feature/memory_model Aug 24, 2023

yyuuttaaoo added this to the v1.8 milestone Sep 26, 2023

yyuuttaaoo added the feature New feature label Sep 26, 2023

Abingcbc deleted the improve-multiline branch November 27, 2024 03:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support continue/end multiple regex patterns #978

feat: support continue/end multiple regex patterns #978

Abingcbc commented Jul 5, 2023 •

edited

Loading

feat: support continue/end multiple regex patterns #978

feat: support continue/end multiple regex patterns #978

Conversation

Abingcbc commented Jul 5, 2023 • edited Loading

配置文件

样例日志

结果

Abingcbc commented Jul 5, 2023 •

edited

Loading