Open Server is my attempt to recreate an OpenAI Compatible Server for generating text, images, embeddings, and storing them in vector databases. It also includes a chat functionality.
The server's request and responses are very similar to OpenAI's API with additional fields needed for different providers. It uses Langchain for the LLM part (Robust and powerful with callbacks) and provider SDKs for image generation and more.
- 9 December 2023 : Added OpenRouter and NeuroAPI to providers as config based on OpenAI provider. Added cost calculation to config and routes in usage.
- Python SDK
- Docker Compose
- ( Soon ) Example Next.JS Front working with Open Server
- Monitoring for LLM generation (LLM Monitor & Prompt Layer)
- ( Soon ) Audio Translation & Generation & Transcription APIs (Local with Whisper, ElevenLabs)
To install the required packages:
pip install -r requirements.txt
To run the server:
python -m openserver.main
Note: For GPU support, you need to manually install pytorch and llama-cpp-python, based on your GPU type (CUDA & ROCM).