A LeapfrogAI API-compatible vllm wrapper for quantized and un-quantized model inferencing across GPU infrastructures.
See the LeapfrogAI documentation website for system requirements and dependencies.
- LeapfrogAI API for a fully RESTful application
The default model that comes with this backend in this repository's officially released images is a 4-bit quantization of the Synthia-7b model.
You can optionally specify different models or quantization types using the following Docker build arguments:
--build-arg HF_HUB_ENABLE_HF_TRANSFER="1"
: Enable or disable HuggingFace Hub transfer (default: 1)--build-arg REPO_ID="TheBloke/Synthia-7B-v2.0-GPTQ"
: HuggingFace repository ID for the model--build-arg REVISION="gptq-4bit-32g-actorder_True"
: Revision or commit hash for the model--build-arg QUANTIZATION="gptq"
: Quantization type (e.g., gptq, awq, or empty for un-quantized)--build-arg TENSOR_PARALLEL_SIZE="1"
: The number of gpus to spread the tensor processing across
To build and deploy the vllm backend Zarf package into an existing UDS Kubernetes cluster:
Important
Execute the following commands from the root of the LeapfrogAI repository
pip install 'huggingface_hub[cli,hf_transfer]' # Used to download the model weights from huggingface
make build-vllm LOCAL_VERSION=dev
uds zarf package deploy packages/vllm/zarf-package-vllm-*-dev.tar.zst --confirm
To run the vllm backend locally:
Important
Execute the following commands from this sub-directory
# Install dev and runtime dependencies
make install
# Clone Model
python src/model_download.py
# Start the model backend
make dev