Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI Failure (key symptom) in RandomNodeOperationsTest.test_node_operations #17739

Closed
vbotbuildovich opened this issue Apr 10, 2024 · 3 comments · Fixed by #18094
Closed

CI Failure (key symptom) in RandomNodeOperationsTest.test_node_operations #17739

vbotbuildovich opened this issue Apr 10, 2024 · 3 comments · Fixed by #18094
Assignees
@mmaslankaprv
Labels
auto-triaged used to know which issues have been opened from a CI job ci-failure team/enterprise helper for jira sync

Comments

@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Apr 10, 2024 •
edited by github-actions bot
Loading

https://buildkite.com/redpanda/vtools/builds/12756

Module: rptest.tests.random_node_operations_test
Class: RandomNodeOperationsTest
Method: test_node_operations
Arguments: {
    "num_to_upgrade": 3,
    "enable_failures": true,
    "with_tiered_storage": true
}

test_id:    RandomNodeOperationsTest.test_node_operations
status:     FAIL
run time:   2017.058 seconds

<BadLogLines nodes=ip-172-31-2-27(1) example="ERROR 2024-04-06 23:52:50,958 [shard 0:admi] admin_api_server - server.cc:649 - [_anonymous] exception intercepted - url: [http://ip-172-31-2-27:9644/v1/brokers/12/decommission] http_return_status[500] reason - seastar::httpd::server_error_exception (Unexpected error: rpc::errc::service_unavailable)">
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 276, in run_test
    return self.test_context.function(self.test)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/mark/_mark.py", line 535, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 177, in wrapped
    redpanda.raise_on_bad_logs(
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1523, in raise_on_bad_logs
    lsearcher.search_logs(_searchable_nodes)
  File "/home/ubuntu/redpanda/tests/rptest/services/utils.py", line 156, in search_logs
    raise BadLogLines(bad_loglines)
rptest.services.utils.BadLogLines: <BadLogLines nodes=ip-172-31-2-27(1) example="ERROR 2024-04-06 23:52:50,958 [shard 0:admi] admin_api_server - server.cc:649 - [_anonymous] exception intercepted - url: [http://ip-172-31-2-27:9644/v1/brokers/12/decommission] http_return_status[500] reason - seastar::httpd::server_error_exception (Unexpected error: rpc::errc::service_unavailable)">

JIRA Link: CORE-2220
@vbotbuildovich vbotbuildovich added auto-triaged used to know which issues have been opened from a CI job ci-failure labels Apr 10, 2024
@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/vtools/builds/12851

@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/redpanda/builds/47861

@dotnwat dotnwat added the team/enterprise helper for jira sync label Apr 18, 2024
@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/vtools/builds/13071

mmaslankaprv added a commit to mmaslankaprv/redpanda that referenced this issue Apr 26, 2024
@mmaslankaprv
admin: fixed mapping of service unavailable error code
1a9f248 
Previously the service_unavailable error code was mapped to generic
unexpected server error. This lead to a situation in which a lot of
errors were logged when broker was restarted as it has an RPC server
running but without the services registered. Changed the mapping of
`rpc::errc::service_unavailable` to HTTP service unavailable error.

Fixes: redpanda-data#17739

Signed-off-by: Michał Maślanka <michal@redpanda.com>
@mmaslankaprv mmaslankaprv mentioned this issue Apr 26, 2024
Merged
7 tasks
@mmaslankaprv mmaslankaprv self-assigned this Apr 26, 2024
vbotbuildovich pushed a commit to vbotbuildovich/redpanda that referenced this issue Apr 26, 2024
@mmaslankaprv @vbotbuildovich
admin: fixed mapping of service unavailable error code
67df5d7 
Previously the service_unavailable error code was mapped to generic
unexpected server error. This lead to a situation in which a lot of
errors were logged when broker was restarted as it has an RPC server
running but without the services registered. Changed the mapping of
`rpc::errc::service_unavailable` to HTTP service unavailable error.

Fixes: redpanda-data#17739

Signed-off-by: Michał Maślanka <michal@redpanda.com>
(cherry picked from commit 1a9f248)
vbotbuildovich pushed a commit to vbotbuildovich/redpanda that referenced this issue Apr 26, 2024
@mmaslankaprv @vbotbuildovich
admin: fixed mapping of service unavailable error code
b2ee61b 
Previously the service_unavailable error code was mapped to generic
unexpected server error. This lead to a situation in which a lot of
errors were logged when broker was restarted as it has an RPC server
running but without the services registered. Changed the mapping of
`rpc::errc::service_unavailable` to HTTP service unavailable error.

Fixes: redpanda-data#17739

Signed-off-by: Michał Maślanka <michal@redpanda.com>
(cherry picked from commit 1a9f248)
This was referenced Apr 26, 2024
Closed
Closed
mmaslankaprv added a commit to vbotbuildovich/redpanda that referenced this issue Apr 30, 2024
@mmaslankaprv
admin: fixed mapping of service unavailable error code
e2a6c89 
Previously the service_unavailable error code was mapped to generic
unexpected server error. This lead to a situation in which a lot of
errors were logged when broker was restarted as it has an RPC server
running but without the services registered. Changed the mapping of
`rpc::errc::service_unavailable` to HTTP service unavailable error.

Fixes: redpanda-data#17739

Signed-off-by: Michał Maślanka <michal@redpanda.com>
(cherry picked from commit 1a9f248)
mmaslankaprv added a commit to vbotbuildovich/redpanda that referenced this issue Apr 30, 2024
@mmaslankaprv
admin: fixed mapping of service unavailable error code
f72d926 
Previously the service_unavailable error code was mapped to generic
unexpected server error. This lead to a situation in which a lot of
errors were logged when broker was restarted as it has an RPC server
running but without the services registered. Changed the mapping of
`rpc::errc::service_unavailable` to HTTP service unavailable error.

Fixes: redpanda-data#17739

Signed-off-by: Michał Maślanka <michal@redpanda.com>
(cherry picked from commit 1a9f248)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mmaslankaprv mmaslankaprv

Labels
auto-triaged used to know which issues have been opened from a CI job ci-failure team/enterprise helper for jira sync
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

admin: fixed mapping of service unavailable error code mmaslankaprv/redpanda
3 participants
@dotnwat @mmaslankaprv @vbotbuildovich