Is it possible to set custom timeout?

[b4git]

I got the following message while working with a slow local model:

Applied edit to index.html
Applied edit to style.css
Request timed out: HTTPConnectionPool(host='localhost', port=1234): Read timed out. (read timeout=600)
Retry in 0.2 seconds.
Commit fc5372c aider: diff --git a/index.js b/index.js

Is it possible to set custom values for the timeout and retry periods?

[paul-gauthier]

Thanks for trying aider!

Aider uses the standard openai python library, which apparently does not accept a timeout parameter.

openai/openai-python#322

[NightOwlCoder]

Unfortunately, it requires a lot of code changes, as it is only available on the beta version:
openai/openai-python#631

---

Looks like timeout is now a possible parameter:

openai/openai-python#322 (comment)

[paul-gauthier]

I will keep an eye on this new version of the api, thanks.

[oreid-zd]

Would be great to add support for a custom timeout.

[paul-gauthier]

What is the error you are facing? The default of 10 minutes is a pretty long timeout.

[Mushoz]

I am also running into timeout issues with my locally hosted QwQ model. It's EXTREMELY chatty, and responses can potentially take quite a long time. Would be great if we could increase the value.

[paul-gauthier]

What is the problem here?

1. Requests are taking longer than the 10 minute default timeout?
2. The default timeout is actually less than 10 minutes?

[Mushoz]

Number 1. I have seen replies close to 20k tokens. At 13ish tokens/seconds that is 25ish minutes. I know it's hardly usable at those speeds, but I find it interesting to see what is possible with today's tech. Furthermore, I do plan on using those long running prompts during breaks on issues that I havent been able to solve with traditional non-COT models.

I would love for that 10 minutes to be configurable, so I can just set it at 1 hour or something ridiculous and roll with it.

[Mushoz]

Example of the issue while benchmarking:

main_model: openai/QwQ-32B-Preview-Q4_K_S
edit_format: architect
fnames: /benchmarks/2024-11-30-20-16-43--QwQ-32B-Preview-Q4_K_S-16k/poker/poker.py
Aider v0.65.2.dev82+ge4a1d6fe
Model: openai/QwQ-32B-Preview-Q4_K_S with architect edit format
Editor model: openai/Qwen2.5-Coder-32B-Instruct-Q4_K_S with whole edit format
Git repo: none
Repo-map: disabled
Added poker.py to the chat.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 72, in map_httpcore_exceptions
    yield
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 236, in handle_request
    resp = self._pool.handle_request(req)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request
    raise exc from None
  File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request
    response = connection.handle_request(
  File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 103, in handle_request
    return self._connection.handle_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 136, in handle_request
    raise exc
  File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 106, in handle_request
    ) = self._receive_response_headers(**kwargs)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 177, in _receive_response_headers
    event = self._receive_event(timeout=timeout)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 217, in _receive_event
    data = self._network_stream.read(
  File "/usr/local/lib/python3.10/site-packages/httpcore/_backends/sync.py", line 126, in read
    with map_exceptions(exc_map):
  File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ReadTimeout: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 991, in _request
    response = self._client.send(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 926, in send
    response = self._send_handling_auth(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 954, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 991, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1027, in _send_single_request
    response = transport.handle_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 235, in handle_request
    with map_httpcore_exceptions():
  File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 89, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/litellm/llms/OpenAI/openai.py", line 860, in completion
    raise e
  File "/usr/local/lib/python3.10/site-packages/litellm/llms/OpenAI/openai.py", line 796, in completion
    self.make_sync_openai_chat_completion_request(
  File "/usr/local/lib/python3.10/site-packages/litellm/llms/OpenAI/openai.py", line 657, in make_sync_openai_chat_completion_request
    raise e
  File "/usr/local/lib/python3.10/site-packages/litellm/llms/OpenAI/openai.py", line 639, in make_sync_openai_chat_completion_request
    raw_response = openai_client.chat.completions.with_raw_response.create(
  File "/usr/local/lib/python3.10/site-packages/openai/_legacy_response.py", line 356, in wrapped
    return cast(LegacyAPIResponse[R], func(*args, **kwargs))
  File "/usr/local/lib/python3.10/site-packages/openai/_utils/_utils.py", line 275, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/openai/resources/chat/completions.py", line 829, in create
    return self._post(
  File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 1278, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
  File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 955, in request
    return self._request(
  File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 1000, in _request
    return self._retry_request(
  File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 1093, in _retry_request
    return self._request(
  File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 1000, in _request
    return self._retry_request(
  File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 1093, in _retry_request
    return self._request(
  File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 1010, in _request
    raise APITimeoutError(request=request) from err
openai.APITimeoutError: Request timed out.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/litellm/main.py", line 1605, in completion
    raise e
  File "/usr/local/lib/python3.10/site-packages/litellm/main.py", line 1578, in completion
    response = openai_chat_completions.completion(
  File "/usr/local/lib/python3.10/site-packages/litellm/llms/OpenAI/openai.py", line 870, in completion
    raise OpenAIError(
litellm.llms.OpenAI.openai.OpenAIError: Request timed out.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/aider/aider/coders/base_coder.py", line 1216, in send_message
    yield from self.send(messages, functions=self.functions)
  File "/aider/aider/coders/base_coder.py", line 1522, in send
    hash_object, completion = send_completion(
  File "/aider/aider/sendchat.py", line 51, in send_completion
    res = litellm.completion(**kwargs)
  File "/usr/local/lib/python3.10/site-packages/litellm/utils.py", line 960, in wrapper
    raise e
  File "/usr/local/lib/python3.10/site-packages/litellm/utils.py", line 849, in wrapper
    result = original_function(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/litellm/main.py", line 3059, in completion
    raise exception_type(
  File "/usr/local/lib/python3.10/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 2136, in exception_type
    raise e
  File "/usr/local/lib/python3.10/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 194, in exception_type
    raise Timeout(
litellm.exceptions.Timeout: litellm.Timeout: APITimeoutError - Request timed out. 
error_str: Request timed out.

litellm.Timeout: APITimeoutError - Request timed out. 
error_str: Request timed out.

[paul-gauthier]

You can try --timeout X.

The change is available in the main branch. You can get it by installing the latest version from github:

aider --install-main-branch

# or...

python -m pip install --upgrade --upgrade-strategy only-if-needed git+https://github.com/Aider-AI/aider.git

If you have a chance to try it, let me know if it works better for you.

[Mushoz]

Wow that was quick, perfect, thanks! The benchmark does not accept the --timeout flag. I am guessing I can add the .aider.conf.yml file in aider's root of the git repository inside the benchmark container, set the timeout value in that file, and then run the benchmark as usual, right?

I am not sure that is working. I have set my openai-api-base and openai-api-key in that same file, to make sure it's really utilizing the file. And I am not able to connect to the specified models unless I explicitely add those to the OPENAI_API_BASE and OPENAI_API_KEY environment variables.

[paul-gauthier]

No, the benchmark is not setup to read yml, etc.

You would need to modify the benchmark code to do:

from aider.llm import litellm
litellm._load_litellm()
litellm._lazy_module.request_timeout = X

[Mushoz]

At the top of the benchmark.py file?

[Mushoz]

At the top of the benchmark.py file?

I tried this, but the timeouts kept on happening. Is there any chance a --timeout switch can be introduced to the benchmark? The benchmark is where this feature is most useful. During normal use, tokens are streamed anyway which I reckon should prevent any timeouts from triggering. It's really the benchmark with the non-streamed responses which desperately need longer timeouts for chatty models such as QwQ

[oreid-zd]

What is the error you are facing? The default of 10 minutes is a pretty long timeout.

Ahh, I might have misinterpreted the purpose of this thread. I was referring specifically to the /web feature. It times out after a few seconds when trying to scrape a webpage.

[paul-gauthier]

I'm going to close this issue for now, but feel free to add a comment here and I will re-open. Or feel free to file a new issue any time.

[Mushoz]

Is there any chance this issue could be revisited to implement this in the benchmark @paul-gauthier ? There have been so many interesting COT models that I would love to benchmark, but the timeout is problematic since the responses can be so long. I want to find out what model would work best, but right now the timeout errors are preventing me from evaluating this properly

[paul-gauthier]

Sorry, I believe the code needed for a timeout is shown above? Just modify benchmark.py.

[Mushoz]

That doesn't work. I have added the following block of code just below the regular imports:

from aider.llm import litellm
litellm._load_litellm()
litellm._lazy_module.request_timeout = 10

This should let it timeout within 10 seconds, which is really easy to test as most queries will take up far longer than 10 seconds. But it will simply continue running past 10 seconds, which means it's probably still using the default timeout value. Could this be a litellm bug @paul-gauthier ?

[vmajor]

I cannot find my bug submission, thus apologies if I am in fact spamming, but I would love to try aider with my local R1, however I cannot due to API timeouts that I cannot circumvent. This is in normal use, not benchmarking, and aider still times out while waiting on R1 to produce anything regardless of the length set with --timeout startup flag.

I need to know whether it is possible to disable any API timeout handling - I (we?) are running a local API anyway so we have direct control over it, ergo any external API 'management' is undesirable.

[paul-gauthier]

I've made some changes for timeouts. You can now set the timeout a few ways:

In code:

from aider import models
models.request_timeout = XXX

In .aider.model.settings.yml:

# global
- name: aider/extra_params
  timeout: XXX

# for a specific model
- name: some/specific-model
  timeout: YYYY

From the command line:

aider --timeout XXX

Note, there seems to be a bug in litellm where it is not honoring the timeout setting for ollama_chat/ models. You may be able to switch to ollama/ to work around this.

BerriAI/litellm#8333

[paul-gauthier]

I'm going to consider this issue solved, and have created a new issue #3161 to track the ollama_chat specific bug.

[paul-gauthier]

@vmajor

The changes are available in the main branch. You can get it by installing the latest version from github:

aider --install-main-branch

# or...

python -m pip install --upgrade --upgrade-strategy only-if-needed git+https://github.com/Aider-AI/aider.git

If you have a chance to try it, let me know if it works better for you.

[github-actions]

I'm closing this bug report since it has been marked as 'fixed' for over 3 weeks. This issue should be resolved in recent versions of aider.

If you find that this bug is still present, please feel free to reopen this issue or create a new one with steps to reproduce.

Note: A bot script made these updates to the issue.

[vmajor]

@vmajor

The changes are available in the main branch. You can get it by installing the latest version from github:

aider --install-main-branch

# or...

python -m pip install --upgrade --upgrade-strategy only-if-needed git+https://github.com/Aider-AI/aider.git

If you have a chance to try it, let me know if it works better for you.

well tried to start it like before:

aider --openai-api-base http://127.0.0.1:8080 --openai-api-key NONE --model deepseek-reasoner --model-metadata-file aider.json --cache-prompts --map-tokens 2048 --timeout 500000000

but got this

litellm.BadRequestError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=deepseek-reasoner
 Pass model as E.g. For 'Huggingface' inference endpoints pass in `completion(model='huggingface/starcoder',..)` Learn more: https://docs.litellm.ai/docs/providers

So now I have to attempt to figure out new configuration settings that will understand that this is a local API. I am not running a standalone LiteLLM, just aider...

ok this was easy, I just added openai, like this openai/deepseek-reasoner

I will next let you know if the timeout issue is resolved

[vmajor]

No timeout yet and it's been 3 hours, so far so good. Timeout is set to 138889 hours, should be enough :D