[14/04/2024 - 19/05/2024] 🐛 💪 Bug Bash \ New Features \ Providers

[krrishdholakia]

Aim: bash 50 bugs this week

• [Bug]: Claude 2 in async stopped working #3027
• [Bug]: Invalid kwarg request used for raising Timeout from a/completion call #3025
• Delete Key not working on Admin UI #3049
• Confusing error messages on failed delete key #3050
• [Feature]: Use litellm with mulitple projects in langfuse #3024
• [Bug]: User UI empty after creating User via API or UI #3021
• [Bug]: Slack Alerting - Repeat budget crossed alerts  #3113
• Langfuse - Allow user to opt into default tags used on Langfuse #3178
• [Bug]: Output provide redis cache password #3060
• Show VertexAI api_base, location and project in Vertex Requests are hanging Alert #3177
• Proxy gemini-pro-vision error: please run pip install Pillow #3051
• Add testing to ensure logging is done to langfuse -> when store_model_in_db=True #3064
• Large number of Run time of job "update_spend (trigger: interval[0:00:07], next run at: 2024-04-16 20:43:59 UTC)" was missed by 0:00:01.947209 in server logs #3073
• Slack alerting don't log entire messages in "responses taking too long" #3137
• Slack alerting on LLM API call failed: 400: {'error': 'Invalid model name passed in'} - show model used #3138
• Give a langfuse Trace id in the slow response and hanging response alerts #3139
• Slack Alerting - embedding requests do not show the input #3195
• [Bug]: is < treated as a special character in azure outputs? #3018
• [Bug]: Cannot delete models in UI and APIs #2951
• [Bug]: Completion breaks if response.model is None #2972
• [Bug]: Empty response when trying to get spend logs #2899
• [Bug]: docker seems to ignore the config file #2889
• [Bug]: Langfuse's cached results are missing info #2885
• [Bug]: S3 Logging seems dead on default_team_settings teams #2852
• [Bug]: The meaning of "max_tokens" reported by /model/info is inconsistent #2834
• [Bug]: Error when passing Function call response with Anthropic on Bedrock - 'ChatCompletionMessageToolCall' object is not subscriptable #2833
• [Bug]: stream_chunk_builder fails to include tool_calls preceded by content #2716
• [Bug]: Function type is None when stream=True for Mistral. #2645
• [Investigate - Potential Bug]: Embedding caching reordering #2990
• [Bug]: AWS Bedrock do not work when using tools call in stream=true mode  #2615
• [Bug]: complete doesn't work for Perplexity models. #2584
• [Bug]: Secure /team/info endpoint #3052
• [Bug]: block key created via ui from creating new keys  #3009
• UI - key info modal shows black screen #3070
• [Bug]: LITELLM_MASTER_KEY leaked to Langfuse #2114
• use tool config for gemini pro function calling #3086
• get_model_info() doesn't seem to accept fully qualified model names like completion() does? #3100
• [Bug]: anthropic streaming, streams chunks all at once #3129
• [Bug]: Gemini fails with "Object of type RepeatedComposite is not JSON serializable" when attempting to call function with array argument #3147
• [Bug]: Incorrect Cohere max_output_tokens values? #3135
• [Bug]: native vertex ai (non-claude) token usage doesn't work for streaming content without if-block. #3155
• [Bug]: llama-3 on replicate is not working as expected #3158
• [Bug]: move replicate calls to native async  #3128
• unit testing on dual cache batch get cache - incorrect mapping of redis response #3181
• redis url ssl fix - support 'redis_url' arg for litellm.router #3182
• [Bug]: Proxy doesn't support Redis with TLS/SSL #3090
• [Bug]: Anthropic: Unable to respond to a tool_use request #3156
• [Bug]: Vertex AI - Function Calling isn't logged properly #3241
• [Bug]: LiteLLM Proxy uses cache for completion even if model is different, no-cache parameter does not work #3199
• [Bug]: Anthropic: Unable to respond to a tool_use request #3156
• [Bug]: chat.completion.chunk's index is invalid when using n>=2 and stream=True #3276
• [Bug]: With a key with access to "All Team Models", list models returns just "all-team-models" #3275
• [Bug]: Router does not immediately fallback when Groq is rate limited #3274
• [Bug]: fix ui - switching routing strategies requires client init #3288
• [Bug]: anthropic stop_sequences: each stop sequence must contain non-whitespace #3286

End-User Cost Tracking

• allow setting end-user budgets #3514
• allow setting Customer credits #3515
• Show user based usage in usage tab UI #3516
• Give monthly/daily breakdown of user spend on UI #3517

New Models

• Google VertexAI imagegen support #3633

New Features

• [Feature]: Support OpenAI Param stream_options={"include_usage": True}  #3490
• [Feature]: Command R & R+ on Amazon Bedrock #3362
• [Feature]: Support OpenAI Param stream_options={"include_usage": True}  #3490
• [Feature]: Return all teams via /team/list endpoint  #3523
• Region based routing (per end-user) #3524
• Support returning streaming token usage across all providers #3553

[krrishdholakia]

link any bugs you find, worth resurfacing this week