Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Serve] serve status breaks when disabling http proxy #41723

Closed
saswatac opened this issue Dec 8, 2023 · 0 comments · Fixed by #42286
Closed

[Serve] serve status breaks when disabling http proxy #41723

saswatac opened this issue Dec 8, 2023 · 0 comments · Fixed by #42286
Assignees
Labels
bug Something that is supposed to be working; but isn't P1 Issue that should be fixed within a few weeks serve Ray Serve Related Issue

Comments

@saswatac
Copy link

saswatac commented Dec 8, 2023

What happened + What you expected to happen

When running serve status, i get an error caused by an validation error.

Traceback (most recent call last):
  File "/home/container_user/.local/bin/serve", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/ray/serve/scripts.py", line 702, in status
    **ServeSubmissionClient(address).get_serve_details()
  File "/usr/local/lib/python3.10/dist-packages/ray/dashboard/modules/serve/sdk.py", line 100, in get_serve_details
    self._raise_error(response)
  File "/usr/local/lib/python3.10/dist-packages/ray/dashboard/modules/dashboard_sdk.py", line 283, in _raise_error
    raise RuntimeError(
RuntimeError: Request failed with status code 503: Failed to get a response from the controller. The GCS may be down, please retry later: ray::ServeController.get_serve_instance_details() (pid=12884, ip=10.244.182.9, actor_id=bea8e193585be32034f33c4001000000, repr=<ray.serve._private.controller.ServeController object at 0x7ff0518c6230>)
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.10/dist-packages/ray/serve/_private/controller.py", line 895, in get_serve_instance_details
    return ServeInstanceDetails(
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for ServeInstanceDetails
proxy_location
  value is not a valid enumeration member; permitted: 'Disabled', 'HeadOnly', 'EveryNode' (type=type_error.enum; enum_values=[<ProxyLocation.Disabled: 'Disabled'>, <ProxyLocation.HeadOnly: 'HeadOnly'>, <ProxyLocation.EveryNode: 'EveryNode'>]).

Versions / Dependencies

2.8.0

Reproduction script

  1. start ray ray start --head
  2. python ray_debug.py
from ray import serve
from ray.serve.config import ProxyLocation

@serve.deployment
class Server:
  def dummy(self):
    print("hello")
    return "hello"

app = Server.bind()
serve.start(detached=True, proxy_location=ProxyLocation.Disabled)
serve.run(app)
  1. run serve status

Issue Severity

Medium: It is a significant difficulty but I can work around it.

@saswatac saswatac added bug Something that is supposed to be working; but isn't triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Dec 8, 2023
@anyscalesam anyscalesam added the serve Ray Serve Related Issue label Dec 11, 2023
@sihanwang41 sihanwang41 added P1 Issue that should be fixed within a few weeks and removed triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Dec 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something that is supposed to be working; but isn't P1 Issue that should be fixed within a few weeks serve Ray Serve Related Issue
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants