Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement OpenAI API Compatibility Layer #89

Open
wants to merge 61 commits into
base: main
Choose a base branch
from
Open

Implement OpenAI API Compatibility Layer #89

wants to merge 61 commits into from

Conversation

jacobthebanana
Copy link
Collaborator

PR Type (Feature)

Short Description

Implement seamless support for the OpenAI API Client without any additional package install on the client side. Streamline researcher and developer experience by abstracting away details about model launching and the SLURM cluster.

Tests Added

web/openai_proxy/test_openai_proxy.py (pytest)

@jacobthebanana
Added OpenAI request forwarding prototype.
5ff1a6b
@jacobthebanana
Added prototype for inspecting OpenAI HTTP API message streams.
e3db91e
@jacobthebanana
Added vector_inference as git submodule.
89bdb84
@jacobthebanana
Implemented auto-scaling and Vector Inference integration.
a75f4ea
@jacobthebanana
Adjusted MODEL_FAMILIES regexp for model variant name.
153ea17
@jacobthebanana
Revised handling of add_backend when automatically scaling up.
f0152d3
@jacobthebanana
Updated vector_inference submodule version.
f7b5122
@jacobthebanana
Updated vector_inference submodule version.
fb74492
@jacobthebanana
Updated vector_inference submodule version.
226a200
@jacobthebanana
Renamed callback url to "vllm_worker_callback"
5f5335c
@jacobthebanana
Made proxy port an environment variable.
e0e4f62
@jacobthebanana
Revised handling of request URL suffix.
b14dcbf
@jacobthebanana
Revised handling of logging in SLURM CLI Launcher.
61c422e
@jacobthebanana
Added heuristics to delete leading model weight folder name
b97e661 
e.g., /model-weights/
@jacobthebanana
Added heuristics to delete leading model weight folder name
6b7ad61 
e.g., /model-weights/
@jacobthebanana
Revised iterator handling in backend de-registering
011306b
@jacobthebanana
Included request statistics info in autoscaling manager log.
78277e6
@jacobthebanana
Revised request counter logic for automatic reset
72504ae
@jacobthebanana
Included preemption info in autoscaling manager log.
3bdf603
@jacobthebanana
Invoke delete-backend command even when backend was already preempted.
abce8b5
@jacobthebanana
Revised automatic scale-up to account for cases where backend was pre…
80b10d8 
…empted during launching.
@jacobthebanana
Implemented Remote SSH executor (not fully tested)
bb8a1f4
@jacobthebanana
Switched vector_inference submodule to origin/develop
21eb4d5
@jacobthebanana
Merged vLLM worker callback into vLLM backend status check.
d982f3f
@jacobthebanana
Added WORKER_SLURM_QOS environment variable.
bd42b36
@jacobthebanana
Applied isort formatting.
b87ce07
@jacobthebanana
Added json-mode flag to Vector Inference commands.
eef0b49
@jacobthebanana
Added instruction about WORKER_SLURM_QOS environment variable.
6b6a6bb
@jacobthebanana
Revised JSON parsing logic in SLURMCLILauncher.
9dce8bd
@jacobthebanana
Revised JSON parsing logic in SLURMCLILauncher.
589c013
@jacobthebanana
Revised get_llm_backend to return LLM backend status whenever possible.
973e93e
@jacobthebanana
Added MIN_UPDATE_INTERVAL_SECONDS and MAX_NUM_HISTORIC_RECORDS env vars.
47a19ca
@jacobthebanana
Revised handling of vector-inference status code other than pending.
663ef73
@jacobthebanana
Revised handling of vector-inference status code other than pending.
b3fca12
@jacobthebanana
Revised handling of vector-inference status code other than pending.
99bc57f
@jacobthebanana
Set default CLI executor to RemoteSSHCommandExecutor.
49c1681
@jacobthebanana
Implemented support for running OpenAI Proxy in Docker
ebf761d
@jacobthebanana
Added /app to PYTHONPATH to enable relative import.
56628eb
@jacobthebanana
Revised Dockerfile to copy "web" folder to Docker image.
99c0f56
@jacobthebanana
Revised Docker-Compose file to enable relative imports.
5621d49
@jacobthebanana
Revised Docker-Compose file to enable relative imports.
45e2347
@jacobthebanana
Bumped base Python image for Flask to 3.11
e5b4293
@jacobthebanana
Updated routes to load env var through shared Config.
4dac9a2
@jacobthebanana
Updated routes to load JOB_SCHEDULER_VENV_PATH through shared Config.
edb0b43
@jacobthebanana
Updated example value for JOB_SCHEDULER_VENV_PATH in shared Config.
dd2dc16
@jacobthebanana
Remote SSH Executor: Revised handling of detached execution.
b2235d5 
Added logging statements.
@jacobthebanana
Remote SSH Executor: Revised logging of CLI args.
b858eb3
@jacobthebanana
Remote SSH Executor: Added logging of CLI response.
088700c
@jacobthebanana
AutoScalingManager: Included _backend_ids_by_model in logging output.
ccf7e8e
@jacobthebanana
SLURMCLILauncher: Revised handling of model_status from Vector Inference
7a79a68
@jacobthebanana
Set gunicorn number of workers to 1.
f496e49
@jacobthebanana
Updated MODEL_FAMILIES.
b46153a
@jacobthebanana
API Gateway: Implemented auto-scaling configs and per-user logging.
b79b82b
@jacobthebanana
API Gateway: Switched to relative imports.
8a1b917 
Added context manager to db_utils log_openai_response.
Moved auto-scaling config examples to web/ and updated gitignore.
@jacobthebanana
Autoscaling: Revised scale-down logic, prioritizing pending backends.
15f1d5b
@jacobthebanana
API Gateway: Updated logging implementation to log only when "usage" …
0c29d46 
…is available.
@jacobthebanana
Autoscaling: Revised scale-up logic to load model names from both req…
176e225 
…uest stats and config.
@jacobthebanana
API Gateway: Implemented JWT integration.
90d2bbb
@jacobthebanana
Implemented base OpenAI Proxy and autoscaling.
b58ecda
@jacobthebanana
Moved autoscaling manager state to DB.
9a3e412
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@jacobthebanana