Sometimes results of some tests disappear if an error occured #115
Labels
bug
Something isn't working
Comments
|
The 'Address already in use' error is likely due to the way test-run assign free ports to tarantool servers. It checks for a free port, then pass it to the tarantool via an environment variable and there is the lag between the check and actual bind. It becomes rarely and have no (known for me) ways to mitigate, because we should save the port in the tarantool_server class to connect to the server to pass lines of a test. Maybe we can read a port tarantool binds from logs and set python's variable then. Anyway, it is not the problem of this issue. The problem is that test-run does not count the test in this case. It should be counted as failed ones. |
Totktonada
added a commit
that referenced
this issue
Sep 5, 2018
It allows to mitigate the error 'Address already in use' while binding admin (console) port. Check [1] for more information (see comment itself, the issue is about the another problem). [1]: #115 (comment)
Totktonada
added a commit
to tarantool/graphql.0
that referenced
this issue
Sep 5, 2018
The problem is that frontend wait infinitely for a storage server, which have not been initialized due to the error like the following. ``` Start failed: builtin/box/console.lua:560: failed to create server localhost:57988: Address already in use ``` The problem is described in [1] (see comment itself, the issue is about the another problem). The gist of the fix is using unix sockets for admin console of non-default servers (storages). The option 'use_unix_sockets' affects only console (admin) socket and does nothing with binary (listen) port of a storage. It is critical in the benchmarking test suite. TBD: update test-run from master after test-run's PR #123. [1]: tarantool/test-run#115 (comment)
Totktonada
added a commit
that referenced
this issue
Sep 5, 2018
It allows to mitigate the error 'Address already in use' while binding admin (console) port. Check [1] for more information (see comment itself, the issue is about the another problem). Unified boolean suite.ini options parsing. [1]: #115 (comment)
Totktonada
added a commit
that referenced
this issue
Sep 5, 2018
It allows to mitigate the error 'Address already in use' while binding admin (console) port. Check [1] for more information (see comment itself, the issue is about the another problem). Unified boolean suite.ini options parsing. [1]: #115 (comment)
Totktonada
added a commit
to tarantool/graphql.0
that referenced
this issue
Sep 5, 2018
The problem is that frontend wait infinitely for a storage server, which have not been initialized due to the error like the following. ``` Start failed: builtin/box/console.lua:560: failed to create server localhost:57988: Address already in use ``` The problem is described in [1] (see comment itself, the issue is about the another problem). The gist of the fix is using unix sockets for admin console of non-default servers (storages), it is why test-run update is needed. The option 'use_unix_sockets' affects only console (admin) socket and does nothing with binary (listen) port of a storage. It is critical in the benchmarking test suite. [1]: tarantool/test-run#115 (comment)
Totktonada
added a commit
to tarantool/graphql.0
that referenced
this issue
Sep 5, 2018
The problem is that frontend wait infinitely for a storage server, which have not been initialized due to the error like the following. ``` Start failed: builtin/box/console.lua:560: failed to create server localhost:57988: Address already in use ``` The problem is described in [1] (see comment itself, the issue is about the another problem). The gist of the fix is using unix sockets for admin console of non-default servers (storages), it is why test-run update is needed. The option 'use_unix_sockets' affects only console (admin) socket and does nothing with binary (listen) port of a storage. It is critical in the benchmarking test suite. [1]: tarantool/test-run#115 (comment)
sergw
pushed a commit
to tarantool/tarantool
that referenced
this issue
Sep 11, 2018
* Allow to use unix socket for non-default servers (tarantool/test-run#115) * Fix inspector error reporting for app tests * Add new config param 'show_reproduce_content' (tarantool/test-run#113) * Print last 15 lines of .result for hung tests (tarantool/test-run#107)
|
Raw idea: choose ports via the coordinator: the main process. |
|
Now we have |
|
Another case: TEST_RUN_TESTS=$(for i in $(seq 1 500); do echo -n "tarantoolctl.test.lua "; done) make testecho $?
0cat test/var/008_app-tap/remote.log
<...>
2019-02-18 15:23:22.934 [27133] main/101/remote tarantoolctl:541 E> Start failed: builtin/box/console.lua:560: failed to create server localhost:44152: Address already in use |
|
Moved out 'address already in use' part into #141. |
|
The problem should be fixed by 7176ced for 'core = tarantool' test suites. I'll propose the fix for 'core = app' tests soon. |
|
The case with tarantoolctl.test.lua is a kinda specific: we check whether a task was reported using its ID (a test name + a configuration). When the same test runs many times; test-run is not able to detect that the test was run less times then should. It should not affect regular testing, so this part of the issue is not important. |
Totktonada
added a commit
that referenced
this issue
Apr 17, 2019
This commit handles the case when test-run found that some of scheduled tasks were not reported as finished by workers. The old behaviour leads to zero exit code in the case when --force option is not set (looks as the unintended mistake). Now this situation causes a testing fail (non-zero exit code) and it does not matter whether --force option is set. Show list of failed tasks always except the case when it is caused by a keyboard interruption. While we are here clarified and changed related test-run messages. All that is about the situation that should never occur. See #115 for such cases.
Totktonada
added a commit
that referenced
this issue
Apr 17, 2019
When test_run:cmd('start server foo') is called from an app test and
tarantool fails to start, the following exception is shown:
| [010] Worker "010_app-tap" received the following error; stopping...
| [010] Traceback (most recent call last):
| [010] File "/home/alex/projects/tarantool-meta/tarantool/test-run/lib/worker.py", line 294, in run_task
| [010] task, self.server, self.inspector)
| [010] File "/home/alex/projects/tarantool-meta/tarantool/test-run/lib/test_suite.py", line 208, in run_test
| [010] short_status = test.run(server)
| [010] File "/home/alex/projects/tarantool-meta/tarantool/test-run/lib/test.py", line 180, in run
| [010] self.execute(server)
| [010] File "/home/alex/projects/tarantool-meta/tarantool/test-run/lib/app_server.py", line 43, in execute
| [010] tarantool.join()
The commit handles this case and the exception should not be shown
anymore.
This commit follows the similar one for tarantool server: 7176ced ('Fix
reporting of non-default server fail at start ').
The reporting of such situations is not ideal for now (we show output
and logs for a default server, but don't do that for non-default one),
but this will be fixed in the scope of #159.
Fixes #115.
Totktonada
added a commit
that referenced
this issue
Apr 29, 2019
This update contains changes from 0.6.5 release (cited below) and usage of yaml.safe_load() instead of yaml.load() in tarantool-python tests (doesn't affect test-run behaviour). The reason why it is updated here is just to keep things in sync and, second, to eliminate usage of yaml.load() w/o an explicit loader everywhere where it is possible. The latter is because it was banned in recent versions of pyyaml in Gentoo Linux; see [1]. [1]: https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=79ba924d94cb0cf8559565178414c2a1d687b90c # tarantool-python 0.6.5 ## Breaking changes This release should not break existing code. ## New features * Added MeshConnection that allows to switch between nodes from a user provided list if a current node is down using round-robin strategy (#106). * Added connection_timeout parameter to Connection (#115). ## Bugfixes * Fixed auto-reconnection in Connection. * Eliminated deprecation warnings on Python 3 (#114). * Added TCP_NODELAY back (it was removed in 0.6.4) (#127). https://github.com/tarantool/tarantool-python/releases/tag/0.6.5
Totktonada
added a commit
that referenced
this issue
Apr 30, 2019
This update contains changes from 0.6.5 release (cited below) and usage of yaml.safe_load() instead of yaml.load() in tarantool-python tests (doesn't affect test-run behaviour). The reason why it is updated here is just to keep things in sync and, second, to eliminate usage of yaml.load() w/o an explicit loader everywhere where it is possible. The latter is because it was banned in recent versions of pyyaml in Gentoo Linux; see [1]. There was also related change 38400e9 ('Update pyyaml version') where yaml.load() was replaced with yaml.safe_load() within test-run itself. [1]: https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=79ba924d94cb0cf8559565178414c2a1d687b90c # tarantool-python 0.6.5 ## Breaking changes This release should not break existing code. ## New features * Added MeshConnection that allows to switch between nodes from a user provided list if a current node is down using round-robin strategy (#106). * Added connection_timeout parameter to Connection (#115). ## Bugfixes * Fixed auto-reconnection in Connection. * Eliminated deprecation warnings on Python 3 (#114). * Added TCP_NODELAY back (it was removed in 0.6.4) (#127). https://github.com/tarantool/tarantool-python/releases/tag/0.6.5
Totktonada
added a commit
that referenced
this issue
Apr 30, 2019
This commit handles the case when test-run found that some of scheduled tasks were not reported as finished by workers. The old behaviour leads to zero exit code in the case when --force option is not set (looks as the unintended mistake). Now this situation causes a testing fail (non-zero exit code) and it does not matter whether --force option is set. Show list of failed tasks always except the case when it is caused by a keyboard interruption. While we are here clarified and changed related test-run messages. All that is about the situation that should never occur. See #115 for such cases.
Totktonada
added a commit
that referenced
this issue
Apr 30, 2019
When test_run:cmd('start server foo') is called from an app test and
tarantool fails to start, the following exception is shown:
| [010] Worker "010_app-tap" received the following error; stopping...
| [010] Traceback (most recent call last):
| [010] File "/home/alex/projects/tarantool-meta/tarantool/test-run/lib/worker.py", line 294, in run_task
| [010] task, self.server, self.inspector)
| [010] File "/home/alex/projects/tarantool-meta/tarantool/test-run/lib/test_suite.py", line 208, in run_test
| [010] short_status = test.run(server)
| [010] File "/home/alex/projects/tarantool-meta/tarantool/test-run/lib/test.py", line 180, in run
| [010] self.execute(server)
| [010] File "/home/alex/projects/tarantool-meta/tarantool/test-run/lib/app_server.py", line 43, in execute
| [010] tarantool.join()
The commit handles this case and the exception should not be shown
anymore.
This commit follows the similar one for tarantool server: 7176ced ('Fix
reporting of non-default server fail at start ').
The reporting of such situations is not ideal for now (we show output
and logs for a default server, but don't do that for non-default one),
but this will be fixed in the scope of #159.
Fixes #115.
Totktonada
added a commit
that referenced
this issue
Apr 30, 2019
This update contains changes from 0.6.5 release (cited below) and usage of yaml.safe_load() instead of yaml.load() in tarantool-python tests (doesn't affect test-run behaviour). The reason why it is updated here is just to keep things in sync and, second, to eliminate usage of yaml.load() w/o an explicit loader everywhere where it is possible. The latter is because it was banned in recent versions of pyyaml in Gentoo Linux; see [1]. There was also related change 38400e9 ('Update pyyaml version') where yaml.load() was replaced with yaml.safe_load() within test-run itself. [1]: https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=79ba924d94cb0cf8559565178414c2a1d687b90c # tarantool-python 0.6.5 ## Breaking changes This release should not break existing code. ## New features * Added MeshConnection that allows to switch between nodes from a user provided list if a current node is down using round-robin strategy (#106). * Added connection_timeout parameter to Connection (#115). ## Bugfixes * Fixed auto-reconnection in Connection. * Eliminated deprecation warnings on Python 3 (#114). * Added TCP_NODELAY back (it was removed in 0.6.4) (#127). https://github.com/tarantool/tarantool-python/releases/tag/0.6.5
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Error:
How to reproduce:
Possible result (should pass 100):
OS:
The text was updated successfully, but these errors were encountered: