English | 中文

AI Gateway

Reliably route to 200+ LLMs with 1 fast & friendly API

The AI Gateway streamlines requests to 250+ language, vision, audio and image models with a unified API. It is production-ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency.

✅  Blazing fast (9.9x faster) with a tiny footprint (~100kb build)
✅  Load balance across multiple models, providers, and keys
✅  Fallbacks make sure your app stays resilient
✅  Automatic Retries with exponential fallbacks come by default
✅  Configurable Request Timeouts to easily handle unresponsive LLM requests
✅  Multimodal to support routing between Vision, TTS, STT, Image Gen, and more models
✅  Plug-in middleware as needed
✅  Battle tested over 480B tokens
✅  Enterprise-ready for enhanced security, scale, and custom deployments

Tip

⭐️ Star this repo to get Github release notifications for new provider integrations and features.

Star History

Setup & Installation

Use the AI gateway through the hosted API or self-host the open-source or enterprise versions on your environment.

👉 Hosted Gateway on portkey.ai (Fastest)

The hosted API is the fastest way to setup an AI Gateway for your Gen AI application. We process billions of tokens daily and is in production with companies like Postman, Haptik, Turing, MultiOn, SiteGPT, and more.

👉 Self-hosting the OSS version (MIT License)

To run the AI gateway locally, execute the following command in your terminal. (Needs npx installed) Or, explore deployment guides for Cloudflare, Docker, Node.js and more here.

npx @portkey-ai/gateway

^{Your AI Gateway is now running on http://localhost:8787 🚀}

👉 Self-hosting the Enterprise Version

The AI Gateway's enterprise version offers enterprise-ready capabilities for org management, governance, security and more out of the box. Compare the open source, hosted and enterprise versions here.

The enterprise deployment architecture, supported platforms is available here - Enterprise Private Cloud Deployments

Making requests through the AI gateway

Compatible with OpenAI API & SDKs

The AI Gateway is compatible with the OpenAI API & SDKs, and extends them to call 200+ LLMs reliably. To use the Gateway through OpenAI, update the client to include the gateway's URL and headers and make requests as usual. The AI gateway can translate requests written in the OpenAI format to the signature expected by the specified provider. View examples

Using the Python SDK   

Portkey Python SDK is a wrapper over the OpenAI Python SDK with added support for additional parameters across all other providers. If you're building with Python, this is the recommended library to connect to the Gateway.

pip install -qU portkey-ai

Using the Node.JS SDK

Portkey JS/TS SDK is a wrapper over the OpenAI JS SDK with added support for additional parameters across all other providers. If you're building with JS or TS, this is the recommended library to connect to the Gateway.

npm install --save portkey-ai

Using the REST APIs

The AI gateway supports OpenAI compatible endpoints with added parameter support for all other providers and models. View API Reference.

Other Integrations

Language	Supported SDKs
JS / TS	LangchainJS LlamaIndex.TS
Python	Langchain LlamaIndex
Go	go-openai
Java	openai-java
Rust	async-openai
Ruby	ruby-openai

Gateway Cookbooks

Trending Cookbooks

• Use models from Nvidia NIM with AI Gateway
• Monitor CrewAI Agents with Portkey!
• Comparing Top 10 LMSYS Models with AI Gateway.

Latest Cookbooks

• Create Synthetic Datasets using Nemotron
• Use Portkey Gateway with Vercel's AI SDK
• Monitor Llama Agents with Portkey

More Examples

Supported Providers

Explore Gateway integrations with 25+ providers and 6+ frameworks.

	Provider	Support	Stream
	OpenAI	✅	✅
	Azure OpenAI	✅	✅
	Anyscale	✅	✅
	Google Gemini & Palm	✅	✅
	Anthropic	✅	✅
	Cohere	✅	✅
	Together AI	✅	✅
	Perplexity	✅	✅
	Mistral	✅	✅
	Nomic	✅	✅
	AI21	✅	✅
	Stability AI	✅	✅
	DeepInfra	✅	✅
	Ollama	✅	✅
	Novita AI	✅	✅

View the complete list of 200+ supported models here

Agents

Gateway seamlessly integrates with popular agent frameworks. Read the documentation here.

Framework	Call 200+ LLMs	Advanced Routing	Caching	Logging & Tracing*	Observability*	Prompt Management*
Autogen	✅	✅	✅	✅	✅	✅
CrewAI	✅	✅	✅	✅	✅	✅
LangChain	✅	✅	✅	✅	✅	✅
Phidata	✅	✅	✅	✅	✅	✅
Llama Index	✅	✅	✅	✅	✅	✅
Control Flow	✅	✅	✅	✅	✅	✅
Build Your Own Agents	✅	✅	✅	✅	✅	✅

*Only available on the hosted app. For detailed documentation click here.

Features

Fallbacks Fallback to another provider or model on failed requests. You can specify the errors on which to trigger the fallback. Improves reliability of your application

Automatic Retries Automatically retry failed requests up to 5 times. An exponential backoff strategy spaces out retry attempts to prevent network overload.

Load Balancing Distribute LLM requests across multiple API keys or AI providers with weights to ensure high availability and optimal performance.

Request Timeouts Manage unruly LLMs & latencies by setting up granular request timeouts, allowing automatic termination of requests that exceed a specified duration.

Multi-modal LLM Gateway Call vision, audio (text-to-speech & speech-to-text), and image generation models from multiple providers — all using the familiar OpenAI signature

Guardrails Verify your LLM inputs AND outputs to adhere to your specified checks. Build your own checks or choose from the 20+ pre-built guardrails.

These features are configured through the Gateway Config added to the x-portkey-config header or the config parameter in the SDKs.

Here's a sample config JSON showcasing the above features. All the features are optional

{
	"retry": { "attempts": 5 },
	"request_timeout": 10000,
	"strategy": { "mode": "fallback" }, // or 'loadbalance', etc
	"targets": [{
		"provider": "openai",
		"api_key": "sk-***"
	},{
		"strategy": {"mode": "loadbalance"}, // Optional nesting
		"targets": {...}
	}]
}

Then use the config in your API requests to the gateway.

Using Gateway Configs

Here's a guide to use the config object in your request.

Gateway Enterprise Version

Make your AI app more reliable and forward compatible, while ensuring complete data security and privacy.

✅  Secure Key Management - for role-based access control and tracking
✅  Simple & Semantic Caching - to serve repeat queries faster & save costs
✅  Access Control & Inbound Rules - to control which IPs and Geos can connect to your deployments
✅  PII Redaction - to automatically remove sensitive data from your requests to prevent inadvertent exposure
✅  SOC2, ISO, HIPAA, GDPR Compliances - for best security practices
✅  Professional Support - along with feature prioritization

Schedule a call to discuss enterprise deployments

Contributing

The easiest way to contribute is to pick an issue with the good first issue tag 💪. Read the contribution guidelines here.

Bug Report? File here | Feature Request? File here

Community

Join our growing community around the world, for help, ideas, and discussions on AI.

• View our official Blog
• Chat with us on Discord
• Follow us on Twitter
• Connect with us on LinkedIn