[QUESTION]: processor_split_log_regex 没有分割行

[liyuefu]

现象: 想用日志[2022-03-09 18:00:00] 来分割.但是没有实现.

[root@rh79 bin]# ./ilogtail
register fun v2 0xa34f3a 0xa34f84 0xa34fda 0xa35574
2022/08/12 10:01:55 DEBUG Now using Go's stdlib log package (via loggers/mappers/stdlib).
load log config /usr/local/ilogtail-1.1.0/bin/plugin_logger.xml
recover stderr
recover stdout

2022-08-12 10:02:34 {"tag:path":"./multiline.log","content":"[2022-03-03 18:00:00] xxx1\nyyyyy\nzzzzzz\n","time":"1660269753"}
2022-08-12 10:02:34 {"tag:path":"./multiline.log","content":"[2022-03-03 18:00:01] xxx2\nyyyyy\nzzzzzz\n","time":"1660269753"}
2022-08-12 10:02:34 {"tag:path":"./multiline.log","content":"[2022-03-03 18:00:00] line1\nyayyyy\nzzzzzz\n","time":"1660269753"}
2022-08-12 10:02:34 {"tag:path":"./multiline.log","content":"[2022-03-03 18:00:01] line2\nyyyyy\nzzzzzz\n","time":"1660269753"}
2022-08-12 10:02:34 {"tag:path":"./multiline.log","content":"[2022-03-03 18:00:00] xxx1\nyyyyy\nzzzzzz\n","time":"1660269753"}
2022-08-12 10:02:34 {"tag:path":"./multiline.log","content":"[2022-03-03 18:00:01] xxx2\nyyyyy\nzzzzzz","time":"1660269753"}
2022-08-12 10:03:13 {"tag:path":"./multiline.log","content":"[2022-03-03 18:00:00] xxx1\nyyyyy\nzzzzzz\n","time":"1660269792"}
2022-08-12 10:03:13 {"tag:path":"./multiline.log","content":"[2022-03-03 18:00:01] xxx2\nyyyyy\nzzzzzz","time":"1660269792"}
2022-08-12 10:04:37 {"tag:path":"./multiline.log","content":"[2022-03-03 18:00:00] xxx1\nyyyyy\nzzzzzz\n*************\nFri Aug 12 09:56:05 2022[2022-03-03 18:00:01] xxx2\nyyyyy\nzzzzzz","time":"1660269875"}
2022-08-12 10:06:01 {"tag:path":"./multiline.log","content":"[2022-03-09 18:00:00]\nFri Aug 12 09:56:05 2022\nARC2 started with pid=20, OS id=12957 \nARC1: Archival started\nARC2: Archival started\nARC1: Becoming the 'no FAL' ARCH\nARC1: Becoming the 'no SRL' ARCH\nARC2: Becoming the heartbeat ARCH\nFri Aug 12 09:56:05 2022\nARC3 started with pid=21, OS id=12959 \nARC3: Archival started\nARC0: STARTING ARCH PROCESSES COMPLETE\n[12951] Successfully onlined Undo Tablespace 2.\nUndo initialization finished serial:0 start:200583234 end:200583274 diff:40 (0 seconds)\nVerifying file header compatibility for 11g tablespace encryption..\nVerifying 11g file header compatibility for tablespace encryption completed\n[2022-03-09 18:00:02]\nQMNC started with pid=22, OS id=12961 \nCompleted: ALTER DATABASE OPEN\nStarting background process CJQ0\nFri Aug 12 09:56:05 2022\nCJQ0 started with pid=24, OS id=12974","time":"1660269958"}

配置文件:

[root@rh79 bin]# cat yaml/file_multiline.yaml
enable: true
inputs:

• Type: file_log
  LogPath: .
  FilePattern: multiline.log
  processors:
• Type: processor_split_log_regex
  SplitRegex: [\d+-\d+-\d+\s\d+:\d+:\d+]\s.*
  SplitKey: content
  PreserveOthers: true
  flushers:
• Type: flusher_stdout
  OnlyStdout: true

测试文本 (oracle 告警日志,手动加入了[2022-03-09 18:00:00] 以分割)

[root@rh79 bin]# cat b.txt > multiline.log
[root@rh79 bin]# cat b.txt
[2022-03-09 18:00:00]
Fri Aug 12 09:56:05 2022
ARC2 started with pid=20, OS id=12957
ARC1: Archival started
ARC2: Archival started
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
ARC2: Becoming the heartbeat ARCH
Fri Aug 12 09:56:05 2022
ARC3 started with pid=21, OS id=12959
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Undo initialization finished serial:0 start:200583234 end:200583274 diff:40 (0 seconds)
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
[2022-03-09 18:00:02]
QMNC started with pid=22, OS id=12961
Completed: ALTER DATABASE OPEN
Starting background process CJQ0
Fri Aug 12 09:56:05 2022
CJQ0 started with pid=24, OS id=12974

[Takuka0311]

针对你的日志格式，可以有如下调整：
解决方法1:在“[2022-03-09 18:00:00]”后增加一个空格
解决方法2:将正则表达式改为“\[\d+-\d+-\d+\s\d+:\d+:\d+]”

[liyuefu]

多谢建议.
我测试了方法2, 当有一次多行写入时,还是有不是以指定格式开头的被当作新的一条记录了.
尤其是当一次又十几行写入一条时.

[Takuka0311]

这里主要是正则表达式的问题。方法1的原理是插件在解析"\[\d+-\d+-\d+\s\d+:\d+:\d+]\s.*"时，"[2022-03-09 18:00:00]"的末尾只有一个换行符，导致插件识别失败，多加一个空格就可以识别成功；方法2则是针对"[2022-03-09 18:00:00]"的格式调整了正则表达式。可以根据自己的数据格式使用合适的正则表达式。

[liyuefu]

非常感谢您的建议.

我其实是想把Oracle的alert日志按照每条分析出来. 每条有多行或者一行. 格式本来不是[2022-03-09 18:00:00], 这是我为了简单测试,加上,模拟文档的环境.
实际数据格式如下:
Fri Aug 12 09:56:05 2022

采用第二种方法时, 有时可以,有时不行.

输出:
这一部分是内容已经在文件里,解析的正确
2022-08-12 17:58:52 {"tag:path":"/u01/app/oracle/diag/rdbms/orcl/orcl/trace/alert_orcl.log","content":"Fri Aug 12 17:56:27 2022\nQMNC started with pid=22, OS id=5654 \nCompleted: ALTER DATABASE OPEN","time":"1660298329"}
2022-08-12 17:58:58 {"tag:path":"/u01/app/oracle/diag/rdbms/orcl/orcl/trace/alert_orcl.log","content":"Fri Aug 12 17:58:58 2022\nThread 1 advanced to log sequence 47 (LGWR switch)\n Current log# 2 seq# 47 mem# 0: /u02/oradata/orcl/redo02.log","time":"1660298338"}
2022-08-12 17:58:58 {"tag:path":"/u01/app/oracle/diag/rdbms/orcl/orcl/trace/alert_orcl.log","content":"Fri Aug 12 17:58:58 2022\nArchived Log entry 45 added for thread 1 sequence 46 ID 0x619df702 dest 1:","time":"1660298338"}

下面部分是实时生成的, Thread 1 advanced ... 这一句被当作单独一句解析了. 上面这个就没有.

!!!
2022-08-12 17:59:28 {"tag:path":"/u01/app/oracle/diag/rdbms/orcl/orcl/trace/alert_orcl.log","content":"Fri Aug 12 17:59:28 2022\nThread 1 cannot allocate new log, sequence 48\nCheckpoint not complete\n Current log# 2 seq# 47 mem# 0: /u02/oradata/orcl/redo02.log","time":"1660298368"}
2022-08-12 17:59:31 {"tag:path":"/u01/app/oracle/diag/rdbms/orcl/orcl/trace/alert_orcl.log","content":"Thread 1 advanced to log sequence 48 (LGWR switch)\n Current log# 1 seq# 48 mem# 0: /u02/oradata/orcl/redo01.log","time":"1660298369"}
2022-08-12 17:59:31 {"tag:path":"/u01/app/oracle/diag/rdbms/orcl/orcl/trace/alert_orcl.log","content":"Fri Aug 12 17:59:29 2022\nArchived Log entry 46 added for thread 1 sequence 47 ID 0x619df702 dest 1:","time":"1660298369"}

我的理解是不是,实时解析时会因为读取内容不确定后面的,所以会判断是独立的.

测试的yaml文件:

^C[root@rh79 bin]# cat yaml/file_multiline.yaml
enable: true
inputs:

• Type: file_log
  LogPath: /u01/app/oracle/diag/rdbms/orcl/orcl/trace
  FilePattern: alert_orcl.log
  processors:
• Type: processor_split_log_regex
  SplitRegex: Fri Aug \d+ \d+:\d+:\d+ \d\d\d\d
  SplitKey: content
  PreserveOthers: true
  flushers:
• Type: flusher_stdout
  OnlyStdout: true
  [root@rh79 bin]#

[messixukejia]

日志多行模式得首先说明下，多行是指一次打印多行，例如java异常调用栈，是一次打印出来的。

从你上面的例子看第一行17:59:28， Thread 1 advanced ... 时间是 17:59:31，中间隔了3s。应该是分两次打印出来的。所以不符合多行模式的范畴。

[messixukejia]

@liyuefu 上述如果解答了你的问题，麻烦标记下mark as answer。
另外，也欢迎将您的使用场景写文章分享出来，也是对社区的贡献，非常期待能得到开发者的反馈。