Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: LogMessageWaitStrategy erroneously assumes each log frame contains a complete log line #5843

Closed
SgtSilvio opened this issue Sep 13, 2022 · 1 comment · Fixed by #5854
Closed

[Bug]: LogMessageWaitStrategy erroneously assumes each log frame contains a complete log line #5843

SgtSilvio opened this issue Sep 13, 2022 · 1 comment · Fixed by #5854
Labels
type/bug

Comments

@SgtSilvio
Copy link
Contributor

SgtSilvio commented Sep 13, 2022 •
edited

Module

Core

Testcontainers version

1.17.3

Using the latest Testcontainers version?

Yes

Host OS

Linux CentOS

Host Arch

x86_64

Docker version

Docker version 20.10.18, build b40c2f6

What happened?

LogMessageWaitStrategy uses a LogContainerCmd to follow logs.
The regex is applied to every OutputFrame that is processed by FrameConsumerResultCallback.
FrameConsumerResultCallback consumes STDOUT or STDERR stream types via processOtherFrame.

Problem:
processOtherFrame passes the content of the com.github.dockerjava.api.model.Frame 1-to-1 to the consumer without doing any processing of log lines (like processRawFrame > normalizeLogLines).
This would only work if the Frame contains a complete log message and not a partial log message.

We reproduced this multiple times. See the log output for one example.

When using a WaitAllStrategy with more LogMessageWaitStrategy, this is more likely to happen because the second log wait strategy only attaches to the log after the first finished, and then of course gets a lot more data at once.

Relevant log output

2022-09-13 00:34:58,297 DEBUG - STDOUT: 2022-09-12 22:34:56,408 INFO  - Extension "HiveMQ Enterprise Extension for Kafka" version 4.9.0-SNAPSHOT started succes
2022-09-13 00:34:58,313 DEBUG - STDOUT: sfully.

from this log statement:

testcontainers-java/core/src/main/java/org/testcontainers/containers/output/WaitingConsumer.java

Line 89 in de1324e

LOGGER.debug("{}: {}", frame.getType(), trimmedFrameText);

Additional Information

No response
@SgtSilvio
Copy link
Contributor Author

Related: #643

This was referenced Sep 18, 2022
Closed
Closed
SgtSilvio added a commit to SgtSilvio/testcontainers-java that referenced this issue Sep 18, 2022
@SgtSilvio
Fixes testcontainers#5843: log consumers are now called with exactly …
3c8eb12 
…one complete log line

Motivation: Docker logs provide a stream, the frames provided are not necessarily split by newline characters. So it might happen that a frame contains partial log lines.
Changes:
- Line splitting, partial log buffering and line merging independently of the stream type (RAW, STDOUT, STDERR)
- OutputFrame does consistently not contain newline characters (independent of TTY)
- ToStringConsumer now adds newlines
- Slf4jLogConsumer does not need to remove any newlines

Also fixes testcontainers#4110, testcontainers#455
@SgtSilvio SgtSilvio mentioned this issue Sep 18, 2022
Merged
SgtSilvio added a commit to SgtSilvio/testcontainers-java that referenced this issue Sep 18, 2022
@SgtSilvio
Fixes testcontainers#5843: log consumers are now called with exactly …
53fa80d 
…one complete log line

Motivation: Docker logs provide a stream, the frames provided are not necessarily split by newline characters. So it might happen that a frame contains partial log lines.
Changes:
- Line splitting, partial log buffering and line merging independently of the stream type (RAW, STDOUT, STDERR)
- OutputFrame does consistently not contain newline characters (independent of TTY)
- ToStringConsumer now adds newlines
- Slf4jLogConsumer does not need to remove any newlines

Also fixes testcontainers#4110, testcontainers#455
SgtSilvio added a commit to SgtSilvio/testcontainers-java that referenced this issue Sep 19, 2022
@SgtSilvio
Fixes testcontainers#5843: log consumers are now called with exactly …
e4842f8 
…one complete log line

Motivation: Docker logs provide a stream, the frames provided are not necessarily split by newline characters. So it might happen that a frame contains partial log lines.
Changes:
- Line splitting, partial log buffering and line merging independently of the stream type (RAW, STDOUT, STDERR)
- OutputFrame does consistently not contain newline characters (independent of TTY)
- ToStringConsumer now adds newlines
- Slf4jLogConsumer does not need to remove any newlines

Also fixes testcontainers#4110, testcontainers#455
SgtSilvio added a commit to SgtSilvio/testcontainers-java that referenced this issue Oct 4, 2022
@SgtSilvio
Fixes testcontainers#5843: log consumers are now called with exactly …
1b43161 
…one complete log line

Motivation: Docker logs provide a stream, the frames provided are not necessarily split by newline characters. So it might happen that a frame contains partial log lines.
Changes:
- Line splitting, partial log buffering and line merging independently of the stream type (RAW, STDOUT, STDERR)
- OutputFrame does consistently not contain newline characters (independent of TTY)
- ToStringConsumer now adds newlines
- Slf4jLogConsumer does not need to remove any newlines

Also fixes testcontainers#4110, testcontainers#455
SgtSilvio added a commit to SgtSilvio/testcontainers-java that referenced this issue Oct 10, 2022
@SgtSilvio
Fixes testcontainers#5843: log consumers are now called with exactly …
f52e799 
…one complete log line

Motivation: Docker logs provide a stream, the frames provided are not necessarily split by newline characters. So it might happen that a frame contains partial log lines.
Changes:
- Line splitting, partial log buffering and line merging independently of the stream type (RAW, STDOUT, STDERR)
- OutputFrame does consistently not contain newline characters (independent of TTY)
- ToStringConsumer now adds newlines
- Slf4jLogConsumer does not need to remove any newlines

Also fixes testcontainers#4110, testcontainers#455
SgtSilvio added a commit to SgtSilvio/testcontainers-java that referenced this issue Oct 12, 2022
@SgtSilvio
Fixes testcontainers#5843: log consumers are now called with exactly …
96df082 
…one complete log line

Motivation: Docker logs provide a stream, the frames provided are not necessarily split by newline characters. So it might happen that a frame contains partial log lines.
Changes:
- Line splitting, partial log buffering and line merging independently of the stream type (RAW, STDOUT, STDERR)
- OutputFrame does consistently not contain newline characters (independent of TTY)
- ToStringConsumer now adds newlines
- Slf4jLogConsumer does not need to remove any newlines

Also fixes testcontainers#4110, testcontainers#455
SgtSilvio added a commit to SgtSilvio/testcontainers-java that referenced this issue Oct 17, 2022
@SgtSilvio
Fixes testcontainers#5843: log consumers are now called with exactly …
74df6d1 
…one complete log line

Motivation: Docker logs provide a stream, the frames provided are not necessarily split by newline characters. So it might happen that a frame contains partial log lines.
Changes:
- Line splitting, partial log buffering and line merging independently of the stream type (RAW, STDOUT, STDERR)
- OutputFrame does consistently not contain newline characters (independent of TTY)
- ToStringConsumer now adds newlines
- Slf4jLogConsumer does not need to remove any newlines

Also fixes testcontainers#4110, testcontainers#455
SgtSilvio added a commit to SgtSilvio/testcontainers-java that referenced this issue Oct 20, 2022
@SgtSilvio
Fixes testcontainers#5843: log consumers are now called with exactly …
342d669 
…one complete log line

Motivation: Docker logs provide a stream, the frames provided are not necessarily split by newline characters. So it might happen that a frame contains partial log lines.
Changes:
- Line splitting, partial log buffering and line merging independently of the stream type (RAW, STDOUT, STDERR)
- OutputFrame does consistently not contain newline characters (independent of TTY)
- ToStringConsumer now adds newlines
- Slf4jLogConsumer does not need to remove any newlines

Also fixes testcontainers#4110, testcontainers#455
SgtSilvio added a commit to SgtSilvio/testcontainers-java that referenced this issue Oct 22, 2022
@SgtSilvio
Fixes testcontainers#5843: log consumers are now called with exactly …
fc4a7f5 
…one complete log line

Motivation: Docker logs provide a stream, the frames provided are not necessarily split by newline characters. So it might happen that a frame contains partial log lines.
Changes:
- Line splitting, partial log buffering and line merging independently of the stream type (RAW, STDOUT, STDERR)
- OutputFrame does consistently not contain newline characters (independent of TTY)
- ToStringConsumer now adds newlines
- Slf4jLogConsumer does not need to remove any newlines

Also fixes testcontainers#4110, testcontainers#455
SgtSilvio added a commit to SgtSilvio/testcontainers-java that referenced this issue Nov 2, 2022
@SgtSilvio
Fixes testcontainers#5843: log consumers are now called with exactly …
164882a 
…one complete log line

Motivation: Docker logs provide a stream, the frames provided are not necessarily split by newline characters. So it might happen that a frame contains partial log lines.
Changes:
- Line splitting, partial log buffering and line merging independently of the stream type (RAW, STDOUT, STDERR)
- OutputFrame does consistently not contain newline characters (independent of TTY)
- ToStringConsumer now adds newlines
- Slf4jLogConsumer does not need to remove any newlines

Also fixes testcontainers#4110, testcontainers#455
SgtSilvio added a commit to SgtSilvio/testcontainers-java that referenced this issue Jan 16, 2023
@SgtSilvio
Fixes testcontainers#5843: log consumers are now called with exactly …
9d79384 
…one complete log line

Motivation: Docker logs provide a stream, the frames provided are not necessarily split by newline characters. So it might happen that a frame contains partial log lines.
Changes:
- Line splitting, partial log buffering and line merging independently of the stream type (RAW, STDOUT, STDERR)
- OutputFrame does consistently not contain newline characters (independent of TTY)
- ToStringConsumer now adds newlines
- Slf4jLogConsumer does not need to remove any newlines

Also fixes testcontainers#4110, testcontainers#455
SgtSilvio added a commit to SgtSilvio/testcontainers-java that referenced this issue Jan 18, 2023
@SgtSilvio
Fixes testcontainers#5843: log consumers are now called with exactly …
15dcf98 
…one complete log line

Motivation: Docker logs provide a stream, the frames provided are not necessarily split by newline characters. So it might happen that a frame contains partial log lines.
Changes:
- Line splitting, partial log buffering and line merging independently of the stream type (RAW, STDOUT, STDERR)
- OutputFrame does consistently not contain newline characters (independent of TTY)
- ToStringConsumer now adds newlines
- Slf4jLogConsumer does not need to remove any newlines

Also fixes testcontainers#4110, testcontainers#455
eddumelendez pushed a commit that referenced this issue Feb 16, 2023
@SgtSilvio
Log consumers are now called with exactly one complete log line (#5854)
3ca5e67 
Docker logs provide a stream, the frames provided are not necessarily split by newline characters. So, it might happen that a frame contains partial log lines.

Changes:
- Line splitting, partial log buffering and line merging independently of the stream type (RAW, STDOUT, STDERR)
- OutputFrame does consistently not contain newline characters (independent of TTY)
- ToStringConsumer now adds newlines
- Slf4jLogConsumer does not need to remove any newlines

Fixes #5843
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug
Projects
None yet
Milestone
No milestone
1 participant
@SgtSilvio