Timeout waiting for request with litellm

[thangld201]

I am using local deployed model with vllm, but faced following error from dspy
File "/home/ubuntu/miniconda3/envs/torch_env/lib/python3.10/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 209, in exception_type raise Timeout( litellm.exceptions.Timeout: litellm.Timeout: APITimeoutError - Request timed out.

As litellm is hardcoded into dspy, I do not see a way (as of now) to change the timeout parameter. Could you help me ?
I'm trying to patch the library and replace litellm with simple request instead, but this takes quite a long time and not very maintainable..

[rohitgarud]

hi @thangld201, I think you first need to find out why it is timing out. Why is vllm taking so much time? Patching should be the very last resort. Also, the dspy.configure and the dspy.LM takes additional arguments in the form of **kwargs, which are passed to litellm call. Find out whats the timeout argument called in litellm and try passing that to dspy.LM

[thangld201]

Hi @rohitgarud, thank you for your fast reply!
The vllm server is working normal, I can infer this from the log. The timeout happens due to either some requests being very long, or something's wrong with litellm, as I met the same problems with litellm-based frameworks in the past. I will try figuring the way out and come back if I find something.