✨ LiteLLM Feb 2025 Roadmap + How to Contribute

[ishaan-jaff]

✨ LiteLLM Feb 2025 Roadmap + How to Contribute

We’re excited for 2025! Below are key improvements with direct links to issues.

Our main focus areas are:

• LLM Translation - Bedrock, OpenAI, Structured Outputs
• Spend Tracking
• Logging (focus on Langfuse)
• Proxy Admin UI Improvements
• General Proxy improvements (endpoints/etc.)
• Security
• Service Availability

What would you like to see added, fixed, or improved in Feb 2025?

Detailed Issues & Feature Requests

How to Contribute (We need help) 🤗🤗

• Pick an Issue: Browse our issue list
• Assign Yourself: Mark yourself as the DRI.
• Resolve: Aim to add 1 unit test + 1 e2e test
• Submit a PR: Open your pull request.

🌟 Goal: Complete 50 items by Feb 28th

---

🔧 LLM Translation

Bedrock

Bugs:

• Titan Embeddings fails with aws_bedrock_runtime_endpoint
• [Bug]: Unable to call tools on bedrock mistral ai through LiteLLM #8434
• Tool calling llama meta.llama3-3-70b-instruct-v1:0 not supported
• Fallbacks not working
• Invalid parameters in Titan requests
• Usage not reported on rerank endpoints
• Token reporting issues
• Wrong toolUse blocks
• 500 vs 400 error for unsupported types
• Image content block issue
• STS credentials not cached
• Proxy error with tool calling

Features:

• /bedrock/invoke/ route support for all Anthropic and Nova models
• Json Mode + vision support for all Anthropic and Nova models on /bedrock/invoke
• Latency-optimized inference support
• Awsume interoperability
• Log Bedrock response headers
• Function calling for Mistral models

---

OpenAI

Bugs:

• Zero usage on streaming completions
• Invalid metadata on API calls
• Temperature not removed for o3-mini
• Inconsistent streaming output
• Missing logprobs in responses
• Text_completion issues
• WebSocket issues in browser
• Audio transcription auth error
• Embedding modality issue

Features:

• Edition methods for Image Generation

---

Anthropic

Bugs:

• Response structure regression (Anthropic)
• Anthropic Citations API Support

---

Vertex AI

Bugs

• Vertex Credential Caching issue

---

New models / providers

• Snowflake Support [Feature]: Support for Snowflake LLMs and Embedding models #7979
• 🚨 Cohere v2 rerank endpoint support s/o @jeromeroussin

General Improvements

• [Bug]: stream_chunk_builder fails to include tool_calls preceded by content #2716

Spend Tracking / Token Counting

Bugs:

• Overcosted cache_hit

Features:

• Track cost for azure data regions
• Cost mapping support
• Add new Export button to get monthly Spend by Organization, Team, Key, Tags. Note: It's important this works when SpendLogs is 1M+ entries
• Realtime API Cost Tracking [Feature]: Realtime API Spend Tracking Support #8410

📊 Logging (focus on Langfuse)

Bugs:

• Logging issues with self-hosted llama
• Langfuse callbacks not executed
• Error in logging handler
• Race condition with Redis caching
• Exception with content_policy_fallbacks
• Missing LANGFUSE_HOST in UI
• Issues with LiteLLM Proxy
• Analytics queue full
• Missing logprobs
• [Bug]: no-log not being respected wth embeddings #8380
• [Bug]: team budget not reseting #8364

Features:

• User info integration
• Multi-Modality & Attachments
• Configurable Trace Parameters
• Return trace_id for failures
• Add trace metadata for rerank
• Support N Choices
• Partial content logging on errors

Proxy Admin UI Improvements

• Ensure viewing Usage works at 1M+ Logs in SpendLogs Table in DB
• Allow Editing Provide API Keys on UI
• Allow testing Cache Health on UI
• Keys Page - delete button is hidden fix sizing

---

General Proxy Improvements

• Informational logging during server startup to help understand what litellm is doing to get ready
• Stop using accept-data-loss for prisma migrations
• Allow backwards compatible prisma migrations

Caching

• Allow namespacing the cache hits by some property of the request? Like a user id or org id. [Feature]: Redis Caching - Allow setting a namespace for redis cache #8411
• Bug fix for using Redis cluster pipeline ops with litellm proxy

🔐 Security

Strengthen system security.

Bugs:

• API Keys visible in debug mode
• [Bug]: Editting a key in the UI clears its user_id via /key/update (vuln?) #8394
• [Bug]: /user/list shows everyone's password #8422
• mask password on /cache/ping endpoint
• [Bug]: Editting a key in the UI clears its user_id via /key/update (vuln?) #8394
• Other security issues

Features:

• Add security headers
• Enforce that all requesting virtual keys follow the sk-{16 chars} format
• Allow rotating master keys

--

---

⚙️ Service Availability

Bugs:

• Infinite loop with background_health_checks
• Infinite loop in TogetherAI params check

---

Adding a New Issue

• File on GitHub: Create an issue.
• Tag Appropriately: Use labels: bedrock, openai, structured outputs, logging, langfuse, security, service availability.

Happy contributing!

[krrishdholakia]

Thank you for writing this @ishaan-jaff 🎉🎉

cc: @vibhavbhat @miraclebakelaser

[krrishdholakia]

cc: @jamesbraza @paul-gauthier @xingyaoww @okhat @joaomdmoura

let us know if we missed anything here

[jannikstdl]

Nice! Is with the Langfuse embedding point also the token monitoring included? Because for example if you start up an embedding model with VLLM, which gives also the tokencount as response I think at the moment LiteLLM is not forwarding them to langfuse.

[krrishdholakia]

Hi @jannikstdl that sounds like a bug. Can you file (or link me) to the ticket which includes the vllm vs. liteLLM response?

[dkindlund]

Note: Next gen wildcard support should include intelligent model type discovery. This is because not every region has every LLM model type available. When given credentials for a specific region, LiteLLM should be able to use provider APIs to intelligently enumerate which model types are available for that region.

Keep in mind, providers (e.g., GCP, AWS) do roll out new support for model types per region every month or so -- so this isn't a static one-time operation. My recommendation would be to have this setup on some sort of monthly cron schedule to regularly refresh this list from time-to-time. Bonus points: If a user in the Admin UI does a "model health" check, that also refreshes this list as well, on-demand.

[krrishdholakia]

I believe we can support this today. We support getting the models from a backend provider for some providers already -

litellm/litellm/llms/fireworks_ai/chat/transformation.py

Line 225 in 64ccf4c

url=f"{api_base}/v1/accounts/{account_id}/models",