whisper

LeapfrogAI Whisper Backend

A LeapfrogAI API-compatible faster-whisper wrapper for audio transcription inferencing across CPU & GPU infrastructures.

Usage

Pre-Requisites

See the LeapfrogAI documentation website for system requirements and dependencies.

Dependent Components

• LeapfrogAI API for a fully RESTful application

Model Selection

See the Deployment section for the CTranslate2 command for pulling and converting a model for inferencing.

Deployment

To build and deploy the whisper backend Zarf package into an existing UDS Kubernetes cluster:

Important

Execute the following commands from the root of the LeapfrogAI repository

pip install 'ctranslate2'          # Used to download and convert the model weights
pip install 'transformers[torch]'  # Used to download and convert the model weights
make build-whisper LOCAL_VERSION=dev
uds zarf package deploy packages/whisper/zarf-package-whisper-*-dev.tar.zst --confirm

Local Development

To run the vllm backend locally without K8s (starting from the root directory of the repository):

# Install dev and runtime dependencies
make install

# Download and convert model
# Change the value for --model to change the whisper base
ct2-transformers-converter --model openai/whisper-base --output_dir .model --copy_files tokenizer.json --quantization float32

# Start the model backend
make dev