Making large AI models cheaper, faster and more accessible

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

A high-throughput and memory-efficient inference and serving engine for LLMs

Faster Whisper transcription with CTranslate2

Large Language Model Text Generation Inference

The Triton Inference Server provides an optimized cloud and edge inferencing solution.

SGLang is a fast serving framework for large language models and vision language models.

💎1MB lightweight face detection model (1MB轻量级人脸检测模型)

Replace OpenAI GPT with another LLM in your app by changing a single line of code. Xinference gives you the freedom to use any LLM you need. With Xinference, you're empowered to run inference with any open-source language models, speech recognition models, and multimodal models, whether in the cloud, on-premises, or even on your laptop.

Adversarial Robustness Toolbox (ART) - Python Library for Machine Learning Security - Evasion, Poisoning, Extraction, Inference - Red and Blue Teams

An easy to use PyTorch to TensorRT converter

An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.

Pre-trained Deep Learning models and demos (high quality and extremely fast)

Sparsity-aware deep learning inference runtime for CPUs

Python Library for learning (Structure and Parameter), inference (Probabilistic and Causal), and simulations in Bayesian Networks.

🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools

MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.

TensorFlow template application for deep learning

PyTorch native quantization and sparsity for training and inference

Efficient, scalable and enterprise-grade CPU/GPU inference server for 🤗 Hugging Face transformer models 🚀