-
-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
LogMessageWaitStrategy for PostgreSQLContainer times out #455
Comments
Is this String not logged on Windows machines? This String should normally be logged two times before the database is ready. |
Yes, it is logged. It already came twice when PostgreSQLContainer starts waiting. |
Ok, so that sounds like some race condition regarding the |
@kiview Would like to ask if one can expect a fix for this bug in the near future? We are developing mostly on Windows machines and heavily using testcontainers in our services. Even if we have a little workaround it would be nice to get this issue solved. Don't know if it is also possible to support you in some way but if, we at least need to get some details explained. :) |
@gbtec-ingogriebsch Since it might involve a race condition (which should be platform independent?), I don't know when it will be fixed. It would help to find this bug on Linux as well. I'll be home next week (and I have a Windows 10 box at home) and I'll try to look into the problem. |
Can someone share a project demonstrating this problem? I just ran I see the following error in my console, the test succeeds however:
I think this happens when the container is shutting down, so nothing too bad. Edit: |
@kiview I'm using Docker for Windows, Version 17.06.1-ce-win24 (13025). I have not yet updated to Version 17.09.0-ce-win32 (13529). We will check if the problem is maybe solved if using the latest docker version. And we can provide a simple test project for the problem we are having at the moment. But it will include nothing else as a Spring Boot app which has configured the postgres testcontainer triggered through the 'special' jdbc url. |
The result of https://github.com/testcontainers/testcontainers-java/blob/master/modules/jdbc-test/src/test/java/org/testcontainers/junit/SimplePostgreSQLTest.java on my machine:
|
It is somewhat environment specific, but seems not to depend on the OS itself. Further investigating. |
After additional research, it seems it is not a race condition, since It might be somehow related to docker-java/docker-java#878, not sure though. |
That's not good. Do you think we should revert use of the log-based wait strategy until we can get to the bottom of this? |
I would appreciate that - we need to upgrade to testcontainers 1.4.x to make use of the latest java docker api and the TC_DAEMON parameter. Reverting it to use |
Yes it would, but that's not really an error or a problem, more like verbose logging stuff happening. Other @rnorth @gbtec-markusmeier |
@kiview Do you have some news for us? Can we provide 'some' information to help you understanding the problem? Or is the next step to follow your suggestion and raise an issue in docker-java? |
@gbtec-ingogriebsch Do you think you can raise a useful issue with enough details in docker-java? If yes this would be great. As I said before, I wasn't really able to reproduce this problem on my machine and pinpoint the differences in the environment, so sadly I'm currently unable to get additional insight into this problem 😞 As a work around you could always reside to extending your own custom |
Just a little heads up: We've currently the problem, that I wonder why this is happening, since we do
|
Just noting that I've had a similar issue with 1.5.1 and the Looking under the hood I can see a netty request of this format: |
Yes, this bug seems a bit more prominent on Windows, but we've observed it on different Linux environments as well. Sadly we couldn't figure out the root cause yet, it might be in the |
@gbtec-ingogriebsch @gbtec-markusmeier |
@kiview Nice stuff! We will try it for sure! Give us a moment, we will come back to you. /CC @gbtec-markusmeier |
Tested with testcontainers:1.7.3:
|
@gbtec-markusmeier Thanks for testing and the feedback! |
Today I got this error too... org.testcontainers.containers.ContainerLaunchException: Container startup failed
at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:214)
at org.testcontainers.containers.GenericContainer.starting(GenericContainer.java:638)
at org.testcontainers.containers.FailureDetectingExternalResource$1.evaluate(FailureDetectingExternalResource.java:29)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:83)
at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:207)
... 9 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:279)
at org.testcontainers.containers.GenericContainer.lambda$start$0(GenericContainer.java:209)
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:76)
... 10 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Timed out waiting for log output matching '.*database system is ready to accept connections.*\s'
at org.testcontainers.containers.wait.strategy.LogMessageWaitStrategy.waitUntilReady(LogMessageWaitStrategy.java:30)
at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:32)
at org.testcontainers.containers.wait.LogMessageWaitStrategy.waitUntilReady(LogMessageWaitStrategy.java:17)
at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:32)
at org.testcontainers.containers.PostgreSQLContainer.waitUntilContainerStarted(PostgreSQLContainer.java:103)
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:258)
... 12 more Probably it helps to localize the problem... I'm not using default Postgres image, but Postgres with PostGIS and Timescale https://hub.docker.com/r/timescale/timescaledb-postgis/ or my own https://hub.docker.com/r/binakot/postgresql-postgis-timescaledb/ For example, when I run the image
For
And for
The most interesting thing that everything worked like a charm before I re-install Windows 10 on my PC 😢 But now It works only with my own image, because:
But how did it work before? I didn't change versions of libraries or images. |
I just tried every version of timescale docker images one by one. |
Maybe we should roll back to the Jdbc based wait strategy for the sake of stability, WDYT @rnorth ? |
Or even better to let to choose the strategy via argument or smth else? |
I've just checked the code and I see that we don't have a Line 115 in c91513d
So one solution would be, to extract a real @gbtec-markusmeier What is your take on this? I know this bug blocks you from upgrading to recent Testcontainers version, so I'd be happy to find a solution. Since I'm a bit uncomfortable with the current state of |
@kiview: In any case we will be happy to be able to have a query based db readiness check again - as this works reliable in our environment. |
After some geek-at-keyboard tests by @gbtec-markusmeier and me, it seems this problem was solved in @gbtec-markusmeier will test a bit more and then we might be able to close this ticket 🙂. |
@gbtec-markusmeier For getting some more confirmation of the underlying bug, could you please try to run 1.9.1 with Netty and check if the problem reappears? Just set
in |
The problem does not occur using netty:
|
Thanks, so probably solved by something else, but solved never the less! 👹 |
testcontainers/testcontainers-java#455 discusses this issue, but they believe the problem was fixed by using netty transport. We're gonna just try one of their other solutions, changing to a HostPortWaitStrategy instead of the default log parsing one.
This problem is very likely caused by #5843 |
…one complete log line Motivation: Docker logs provide a stream, the frames provided are not necessarily split by newline characters. So it might happen that a frame contains partial log lines. Changes: - Line splitting, partial log buffering and line merging independently of the stream type (RAW, STDOUT, STDERR) - OutputFrame does consistently not contain newline characters (independent of TTY) - ToStringConsumer now adds newlines - Slf4jLogConsumer does not need to remove any newlines Also fixes testcontainers#4110, testcontainers#455
…one complete log line Motivation: Docker logs provide a stream, the frames provided are not necessarily split by newline characters. So it might happen that a frame contains partial log lines. Changes: - Line splitting, partial log buffering and line merging independently of the stream type (RAW, STDOUT, STDERR) - OutputFrame does consistently not contain newline characters (independent of TTY) - ToStringConsumer now adds newlines - Slf4jLogConsumer does not need to remove any newlines Also fixes testcontainers#4110, testcontainers#455
…one complete log line Motivation: Docker logs provide a stream, the frames provided are not necessarily split by newline characters. So it might happen that a frame contains partial log lines. Changes: - Line splitting, partial log buffering and line merging independently of the stream type (RAW, STDOUT, STDERR) - OutputFrame does consistently not contain newline characters (independent of TTY) - ToStringConsumer now adds newlines - Slf4jLogConsumer does not need to remove any newlines Also fixes testcontainers#4110, testcontainers#455
…one complete log line Motivation: Docker logs provide a stream, the frames provided are not necessarily split by newline characters. So it might happen that a frame contains partial log lines. Changes: - Line splitting, partial log buffering and line merging independently of the stream type (RAW, STDOUT, STDERR) - OutputFrame does consistently not contain newline characters (independent of TTY) - ToStringConsumer now adds newlines - Slf4jLogConsumer does not need to remove any newlines Also fixes testcontainers#4110, testcontainers#455
…one complete log line Motivation: Docker logs provide a stream, the frames provided are not necessarily split by newline characters. So it might happen that a frame contains partial log lines. Changes: - Line splitting, partial log buffering and line merging independently of the stream type (RAW, STDOUT, STDERR) - OutputFrame does consistently not contain newline characters (independent of TTY) - ToStringConsumer now adds newlines - Slf4jLogConsumer does not need to remove any newlines Also fixes testcontainers#4110, testcontainers#455
…one complete log line Motivation: Docker logs provide a stream, the frames provided are not necessarily split by newline characters. So it might happen that a frame contains partial log lines. Changes: - Line splitting, partial log buffering and line merging independently of the stream type (RAW, STDOUT, STDERR) - OutputFrame does consistently not contain newline characters (independent of TTY) - ToStringConsumer now adds newlines - Slf4jLogConsumer does not need to remove any newlines Also fixes testcontainers#4110, testcontainers#455
…one complete log line Motivation: Docker logs provide a stream, the frames provided are not necessarily split by newline characters. So it might happen that a frame contains partial log lines. Changes: - Line splitting, partial log buffering and line merging independently of the stream type (RAW, STDOUT, STDERR) - OutputFrame does consistently not contain newline characters (independent of TTY) - ToStringConsumer now adds newlines - Slf4jLogConsumer does not need to remove any newlines Also fixes testcontainers#4110, testcontainers#455
…one complete log line Motivation: Docker logs provide a stream, the frames provided are not necessarily split by newline characters. So it might happen that a frame contains partial log lines. Changes: - Line splitting, partial log buffering and line merging independently of the stream type (RAW, STDOUT, STDERR) - OutputFrame does consistently not contain newline characters (independent of TTY) - ToStringConsumer now adds newlines - Slf4jLogConsumer does not need to remove any newlines Also fixes testcontainers#4110, testcontainers#455
…one complete log line Motivation: Docker logs provide a stream, the frames provided are not necessarily split by newline characters. So it might happen that a frame contains partial log lines. Changes: - Line splitting, partial log buffering and line merging independently of the stream type (RAW, STDOUT, STDERR) - OutputFrame does consistently not contain newline characters (independent of TTY) - ToStringConsumer now adds newlines - Slf4jLogConsumer does not need to remove any newlines Also fixes testcontainers#4110, testcontainers#455
…one complete log line Motivation: Docker logs provide a stream, the frames provided are not necessarily split by newline characters. So it might happen that a frame contains partial log lines. Changes: - Line splitting, partial log buffering and line merging independently of the stream type (RAW, STDOUT, STDERR) - OutputFrame does consistently not contain newline characters (independent of TTY) - ToStringConsumer now adds newlines - Slf4jLogConsumer does not need to remove any newlines Also fixes testcontainers#4110, testcontainers#455
…one complete log line Motivation: Docker logs provide a stream, the frames provided are not necessarily split by newline characters. So it might happen that a frame contains partial log lines. Changes: - Line splitting, partial log buffering and line merging independently of the stream type (RAW, STDOUT, STDERR) - OutputFrame does consistently not contain newline characters (independent of TTY) - ToStringConsumer now adds newlines - Slf4jLogConsumer does not need to remove any newlines Also fixes testcontainers#4110, testcontainers#455
…one complete log line Motivation: Docker logs provide a stream, the frames provided are not necessarily split by newline characters. So it might happen that a frame contains partial log lines. Changes: - Line splitting, partial log buffering and line merging independently of the stream type (RAW, STDOUT, STDERR) - OutputFrame does consistently not contain newline characters (independent of TTY) - ToStringConsumer now adds newlines - Slf4jLogConsumer does not need to remove any newlines Also fixes testcontainers#4110, testcontainers#455
When waiting for Postgres to start up, PostgreSQLContainer looks for the String
".*database system is ready to accept connections.*\\s"
.The database though has already started when PostgreSQLContainer is ready for waiting and thus will never find the String.
This issues seems to appear only on Windows machines - on Linux it was not observable.
The new PostgreSQLContainer wait behaviour was introduced in #327.
The text was updated successfully, but these errors were encountered: