Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[self-hosted] Microceph setup fixture often fails with: "error to connect to http://localhost/testbucket" #171

Closed
phvalguima opened this issue May 5, 2024 · 3 comments · Fixed by #175
Closed

[self-hosted] Microceph setup fixture often fails with: "error to connect to http://localhost/testbucket" #171

phvalguima opened this issue May 5, 2024 · 3 comments · Fixed by #175
Labels
bug Something isn't working

Comments

@phvalguima
Copy link
Contributor

We have two types of backup tests: 2x runners as self-hosted (bigger VMs) to run the large-deployments; and 2x runners as GH-hosted for the small-deployments in OpenSearch.

The self-hosted runners are often failing in the microceph setup. One example: https://github.com/canonical/opensearch-operator/actions/runs/8959102660/job/24604651827

The error rate is often enough (last 2x runs I executed, 3 of 4 tests for backup with self-hosted failed with this error).

Given this error is only observed on self-hosted, I believe it has to do with the NO_PROXY settings. Generally, we must have in our no-proxy setup something like:

# If we already have some content on NO_PROXY
export NO_PROXY=$NO_PROXY,::1,127.0.0.1,localhost

This ensures the host will not forward local-bound traffic to the proxy and instead resolve within the host.

The full stack trace is:

Traceback (most recent call last):
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/httpsession.py", line 464, in send
    urllib_response = conn.urlopen(
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/urllib3/connectionpool.py", line 799, in urlopen
    retries = retries.increment(
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/urllib3/util/retry.py", line 525, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/awsrequest.py", line 96, in request
    rval = super().request(method, url, body, headers, *args, **kwargs)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/urllib3/connection.py", line 244, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.10/http/client.py", line 1283, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1329, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/awsrequest.py", line 123, in _send_output
    self.send(msg)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/awsrequest.py", line 223, in send
    return super().send(str)
  File "/usr/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPConnection object at 0x71360e1e78e0>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 341, in from_call
    result: Optional[TResult] = func()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 241, in <lambda>
    lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 85, in pytest_runtest_setup
    yield from unraisable_exception_runtest_hook()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 65, in unraisable_exception_runtest_hook
    yield
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/logging.py", line 844, in pytest_runtest_setup
    yield from self._runtest_for(item, "setup")
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/logging.py", line 833, in _runtest_for
    yield
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/capture.py", line 873, in pytest_runtest_setup
    return (yield)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/threadexception.py", line 82, in pytest_runtest_setup
    yield from thread_exception_runtest_hook()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/threadexception.py", line 63, in thread_exception_runtest_hook
    yield
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 159, in pytest_runtest_setup
    item.session._setupstate.setup(item)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 515, in setup
    raise exc
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 512, in setup
    col.setup()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/python.py", line 1630, in setup
    self._request._fillfixtures()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 683, in _fillfixtures
    item.funcargs[argname] = self.getfixturevalue(argname)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 518, in getfixturevalue
    fixturedef = self._get_active_fixturedef(argname)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 603, in _get_active_fixturedef
    fixturedef.execute(request=subrequest)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 1035, in execute
    fixturedef = request._get_active_fixturedef(argname)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 603, in _get_active_fixturedef
    fixturedef.execute(request=subrequest)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 1035, in execute
    fixturedef = request._get_active_fixturedef(argname)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 603, in _get_active_fixturedef
    fixturedef.execute(request=subrequest)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 1053, in execute
    raise exc
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 341, in from_call
    result: Optional[TResult] = func()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 241, in <lambda>
    lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 85, in pytest_runtest_setup
    yield from unraisable_exception_runtest_hook()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 65, in unraisable_exception_runtest_hook
    yield
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/logging.py", line 844, in pytest_runtest_setup
    yield from self._runtest_for(item, "setup")
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/logging.py", line 833, in _runtest_for
    yield
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/capture.py", line 873, in pytest_runtest_setup
    return (yield)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/threadexception.py", line 82, in pytest_runtest_setup
    yield from thread_exception_runtest_hook()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/threadexception.py", line 63, in thread_exception_runtest_hook
    yield
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 159, in pytest_runtest_setup
    item.session._setupstate.setup(item)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 515, in setup
    raise exc
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 512, in setup
    col.setup()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/python.py", line 1630, in setup
    self._request._fillfixtures()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 683, in _fillfixtures
    item.funcargs[argname] = self.getfixturevalue(argname)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 518, in getfixturevalue
    fixturedef = self._get_active_fixturedef(argname)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 603, in _get_active_fixturedef
    fixturedef.execute(request=subrequest)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 1035, in execute
    fixturedef = request._get_active_fixturedef(argname)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 603, in _get_active_fixturedef
    fixturedef.execute(request=subrequest)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 1035, in execute
    fixturedef = request._get_active_fixturedef(argname)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 603, in _get_active_fixturedef
    fixturedef.execute(request=subrequest)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 1053, in execute
    raise exc
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 341, in from_call
    result: Optional[TResult] = func()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 241, in <lambda>
    lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 85, in pytest_runtest_setup
    yield from unraisable_exception_runtest_hook()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 65, in unraisable_exception_runtest_hook
    yield
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/logging.py", line 844, in pytest_runtest_setup
    yield from self._runtest_for(item, "setup")
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/logging.py", line 833, in _runtest_for
    yield
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/capture.py", line 873, in pytest_runtest_setup
    return (yield)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/threadexception.py", line 82, in pytest_runtest_setup
    yield from thread_exception_runtest_hook()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/threadexception.py", line 63, in thread_exception_runtest_hook
    yield
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 159, in pytest_runtest_setup
    item.session._setupstate.setup(item)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 515, in setup
    raise exc
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 512, in setup
    col.setup()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/python.py", line 1630, in setup
    self._request._fillfixtures()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 683, in _fillfixtures
    item.funcargs[argname] = self.getfixturevalue(argname)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 518, in getfixturevalue
    fixturedef = self._get_active_fixturedef(argname)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 603, in _get_active_fixturedef
    fixturedef.execute(request=subrequest)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 1035, in execute
    fixturedef = request._get_active_fixturedef(argname)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 603, in _get_active_fixturedef
    fixturedef.execute(request=subrequest)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 1035, in execute
    fixturedef = request._get_active_fixturedef(argname)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 603, in _get_active_fixturedef
    fixturedef.execute(request=subrequest)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 1073, in execute
    result = ihook.pytest_fixture_setup(fixturedef=self, request=request)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 182, in _multicall
    return outcome.get_result()
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_result.py", line 100, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 167, in _multicall
    teardown.throw(outcome._exception)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/setuponly.py", line 36, in pytest_fixture_setup
    return (yield)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 1122, in pytest_fixture_setup
    result = call_fixture_func(fixturefunc, request, kwargs)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/_pytest/fixtures.py", line 884, in call_fixture_func
    fixture_result = fixturefunc(**kwargs)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/pytest_microceph/_plugin.py", line 46, in microceph
    boto3.client(
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/client.py", line [565](https://github.com/canonical/opensearch-operator/actions/runs/8959102660/job/24604651827#step:22:566), in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/client.py", line 1001, in _make_api_call
    http, parsed_response = self._make_request(
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/client.py", line 1027, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/endpoint.py", line 119, in make_request
    return self._send_request(request_dict, operation_model)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/endpoint.py", line 202, in _send_request
    while self._needs_retry(
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/endpoint.py", line 354, in _needs_retry
    responses = self._event_emitter.emit(
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/hooks.py", line 412, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/hooks.py", line 256, in emit
    return self._emit(event_name, kwargs)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/hooks.py", line 239, in _emit
    response = handler(**kwargs)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/retryhandler.py", line 207, in __call__
    if self._checker(**checker_kwargs):
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/retryhandler.py", line 284, in __call__
    should_retry = self._should_retry(
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/retryhandler.py", line 320, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/retryhandler.py", line 363, in __call__
    checker_response = checker(
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/retryhandler.py", line 247, in __call__
    return self._check_caught_exception(
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/retryhandler.py", line 416, in _check_caught_exception
    raise caught_exception
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/endpoint.py", line 281, in _do_get_response
    http_response = self._send(request)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/endpoint.py", line 377, in _send
    return self.http_session.send(request)
  File "/home/ubuntu/github-runner/_work/opensearch-operator/opensearch-operator/.tox/integration/lib/python3.10/site-packages/botocore/httpsession.py", line 493, in send
    raise EndpointConnectionError(endpoint_url=request.url, error=e)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://localhost/testbucket"
@github-actions GitHub Actions
Copy link

github-actions bot commented May 5, 2024

https://warthogs.atlassian.net/browse/DPE-4289

@carlcsaposs-canonical carlcsaposs-canonical added the bug Something isn't working label May 6, 2024
@carlcsaposs-canonical
Copy link
Contributor

Posting in https://chat.canonical.com/canonical/channels/github-actions-self-hosted-runners, since it appears to be an issue with self-hosted runners

I believe aproxy is supposed to be configured on the self-hosted runners so that no changes are needed on our end with regards to the proxy

@carlcsaposs-canonical
Copy link
Contributor

fyi: https://github.com/canonical/data-platform-workflows/blob/main/.github/workflows/integration_test_charm.md#user-content-fn-1-bccca2791f328953660bf6ef3bd00a3e

phvalguima added a commit that referenced this issue May 7, 2024
@phvalguima
[DPE-4289] Add microceph check
34e44c4 
Check the microceph status before continuing with the fixture code.

Closes #171
@phvalguima phvalguima mentioned this issue May 7, 2024
Closed
carlcsaposs-canonical added a commit that referenced this issue May 8, 2024
@carlcsaposs-canonical
patch(pytest_plugins/microceph): Add retry if microceph not ready
661871e 
Fixes #171 (on self-hosted runners, microceph is not ready before we create bucket)
Merged
carlcsaposs-canonical added a commit that referenced this issue May 14, 2024
@carlcsaposs-canonical
patch(pytest_plugins/microceph): Add retry if microceph not ready (#175)
e7c618e 
Fixes #171 (on self-hosted runners, microceph is not ready before we
create bucket)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
2 participants
@phvalguima @carlcsaposs-canonical