Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

google.api_core.exceptions.RetryError: Deadline of 60.0s exceeded while calling target function, last exception: 503 Getting metadata from plugin failed with error: ('invalid_grant: Bad Request', {'error': 'invalid_grant', 'error_description': 'Bad Request'}) #117

Closed
kyegomez opened this issue Dec 14, 2023 · 21 comments
Labels
component:python sdk Issue/PR related to Python SDK status:awaiting user response Awaiting a response from the author status:stale Issue/PR will be closed automatically if there's no further activity type:bug Something isn't working

Comments

@kyegomez
Copy link

Description of the bug:

package verison: 0.3.1

  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/api_core/grpc_helpers.py", line 65, in error_remapped_callable
    return callable_(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/grpc/_channel.py", line 1161, in __call__
    return _end_unary_response_blocking(state, call, False, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/grpc/_channel.py", line 1004, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "Getting metadata from plugin failed with error: ('invalid_grant: Bad Request', {'error': 'invalid_grant', 'error_description': 'Bad Request'})"
        debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2023-12-13T23:10:10.374012-08:00", grpc_status:14, grpc_message:"Getting metadata from plugin failed with error: (\'invalid_grant: Bad Request\', {\'error\': \'invalid_grant\', \'error_description\': \'Bad Request\'})"}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/api_core/retry.py", line 191, in retry_target
    return target()
           ^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.ServiceUnavailable: 503 Getting metadata from plugin failed with error: ('invalid_grant: Bad Request', {'error': 'invalid_grant', 'error_description': 'Bad Request'})

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/defalt/Desktop/Athena/research/swarms/gemini.py", line 23, in <module>
    response = model.generate_content("The opposite of hot is")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/generativeai/generative_models.py", line 243, in generate_content
    response = self._client.generate_content(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/ai/generativelanguage_v1beta/services/generative_service/client.py", line 566, in generate_content
    response = rpc(
               ^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/api_core/gapic_v1/method.py", line 113, in __call__
    return wrapped_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/api_core/retry.py", line 349, in retry_wrapped_func
    return retry_target(
           ^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/api_core/retry.py", line 207, in retry_target
    raise exceptions.RetryError(
google.api_core.exceptions.RetryError: Deadline of 60.0s exceeded while calling target function, last exception: 503 Getting metadata from plugin failed with error: ('invalid_grant: Bad Request', {'error': 'invalid_grant', 'error_description': 'Bad Request'})

Actual vs expected behavior:

No response

Any other information you'd like to share?

No response

@kyegomez kyegomez added component:python sdk Issue/PR related to Python SDK type:bug Something isn't working labels Dec 14, 2023
@huqin-RM
Copy link

I also encountered this problem GRPC returns 503, but my error is google.api_core.exceptions.RetryError: Deadline of 60.0s exceeded while calling target function, last exception: 503 failed to connect to all addresses; last error: UNKNOWN: ipv4 :172.217.160.74:443: Failed to connect to remote host: FD Shutdown. IP cannot be accessed

@huqin-RM
Copy link

20231214-165300

@HaoyuanShi
Copy link

The same problem encountered by me

@huqin-RM
Copy link

我也遇到了这个问题 GRPC 返回 503,但我的错误是 google.api_core.exceptions.RetryError: Deadline of 60.0s超过调用目标函数时,最后一个异常:503 无法连接到所有地址;最后一个错误:未知:ipv4:172.217.160.74:443:无法连接到远程主机:FD 关闭。 IP无法访问

You can bypass GRPC and use rest:

genai.configure(api_key=GOOGLE_API_KEY,transport='rest')

@kyegomez
Copy link
Author

@huqin-RM thanks that worked, I'm integrating it into the Swarms framework that enables developers to Build, Deploy, and Scale Reliable Autonomous Agents!

https://www.swarms.world/

And, here is the docs for our abstracted away Gemini wrapper:
https://swarms.apac.ai/en/latest/swarms/models/gemini/

@sleeper1023
Copy link

我也遇到了这个问题 GRPC 返回 503,但我的错误是 google.api_core.exceptions.RetryError: Deadline of 60.0s超过调用目标函数时,最后一个异常:503 无法连接到所有地址;最后一个错误:未知:ipv4:172.217.160.74:443:无法连接到远程主机:FD 关闭。 IP无法访问

You can bypass GRPC and use rest:

genai.configure(api_key=GOOGLE_API_KEY,transport='rest')

This works, thank you very much

@HaoyuanShi
Copy link

我也遇到了这个问题 GRPC 返回 503,但我的错误是 google.api_core.exceptions.RetryError: Deadline of 60.0s超过调用目标函数时,最后一个异常:503 无法连接到所有地址;最后一个错误:未知:ipv4:172.217.160.74:443:无法连接到远程主机:FD 关闭。 IP无法访问

You can bypass GRPC and use rest:

genai.configure(api_key=GOOGLE_API_KEY,transport='rest')

It really works.

@RickYangzz
Copy link

我也遇到了这个问题 GRPC 返回 503,但我的错误是 google.api_core.exceptions.RetryError: Deadline of 60.0s超过调用目标函数时,最后一个异常:503 无法连接到所有地址;最后一个错误:未知:ipv4:172.217.160.74:443:无法连接到远程主机:FD 关闭。 IP无法访问

You can bypass GRPC and use rest:

genai.configure(api_key=GOOGLE_API_KEY,transport='rest')

Thank you, it's worked. And I still want to know why you know this way can work?

@huqin-RM
Copy link

我也遇到了这个问题 GRPC Return 503,但我的错误是 google.api_core.exceptions.RetryError: Deadline of 60.0s 超过调用目标函数时,最后一个异常:503 无法连接到所有地址;最后一个错误:未知: ipv4:172.217.160.74:443:无法连接到远程主机:FD关闭。IP无法访问

您可以绕过 GRPC 并使用休息:
genai.configure(api_key=GOOGLE_API_KEY,transport=“休息”)

谢谢,已经成功了。我还想知道为什么你知道这种方法可行?
Because the example about REST in the official manual is to directly use curl to make API requests. I can run the command line directly from my terminal, so I'm sure it's not a network problem.
Looking further at the pythonsdk source code, there are the following comments:
20231218-184550

@RickYangzz
Copy link

我也遇到了这个问题 GRPC Return 503,但我的错误是 google.api_core.exceptions.RetryError: Deadline of 60.0s 超过调用目标函数时,最后一个异常:503 无法连接到所有地址;最后一个错误:未知: ipv4:172.217.160.74:443:无法连接到远程主机:FD关闭。IP无法访问

您可以绕过 GRPC 并使用休息:
genai.configure(api_key=GOOGLE_API_KEY,transport=“休息”)

谢谢,已经成功了。我还想知道为什么你知道这种方法可行?
Because the example about REST in the official manual is to directly use curl to make API requests. I can run the command line directly from my terminal, so I'm sure it's not a network problem.
Looking further at the pythonsdk source code, there are the following comments:
20231218-184550

Thank you. You have a clear way to solve problem.

@cgnannan
Copy link

我也遇到了这个问题 GRPC Return 503,但我的错误是 google.api_core.exceptions.RetryError: Deadline of 60.0s 超过调用目标函数时,最后一个异常:503 无法连接到所有地址;最后一个错误:未知: ipv4:172.217.160.74:443:无法连接到远程主机:FD关闭。IP无法访问

您可以绕过 GRPC 并使用休息:

genai.configure(api_key=GOOGLE_API_KEY,transport=“休息”)

谢谢,已经成功了。我还想知道为什么你知道这种方法可行?

Because the example about REST in the official manual is to directly use curl to make API requests. I can run the command line directly from my terminal, so I'm sure it's not a network problem.

Looking further at the pythonsdk source code, there are the following comments:

20231218-184550

Thanks a lot.It works and solves a problem that has been bothering me for three weeks.

@OhMyDamnCode
Copy link

OhMyDamnCode commented Jan 18, 2024

我也遇到了这个问题 GRPC 返回 503,但我的错误是 google.api_core.exceptions.RetryError: Deadline of 60.0s超过调用目标函数时,最后一个异常:503 无法连接到所有地址;最后一个错误:未知:ipv4:172.217.160.74:443:无法连接到远程主机:FD 关闭。 IP无法访问

You can bypass GRPC and use rest:

genai.configure(api_key=GOOGLE_API_KEY,transport='rest')

I/m having this error msg now:
Traceback (most recent call last):
File "/Users/hulksi/Desktop/upload genreList/update(genreList).py", line 13, in
genai.configure(api_key=GOOGLE_API_KEY, transport='rest')
AttributeError: module 'genai' has no attribute 'configure'

@bezir
Copy link

bezir commented Feb 8, 2024

In my case the problem was docker, I reinstalled the docker to fix the problem (though I acknowledge there might be a better solution)

@HrushikeshPawar
Copy link

我也遇到了这个问题 GRPC 返回 503,但我的错误是 google.api_core.exceptions.RetryError: Deadline of 60.0s超过调用目标函数时,最后一个异常:503 无法连接到所有地址;最后一个错误:未知:ipv4:172.217.160.74:443:无法连接到远程主机:FD 关闭。 IP无法访问

You can bypass GRPC and use rest:

genai.configure(api_key=GOOGLE_API_KEY,transport='rest')

The problem with GRPC is still same even after two months.
But this transport='rest' does solve the problem. Thanks

Hoping that the problem with GRPC will be solved soon. Facing same issue with TextGeneration on VertexAI as well.

@Hoper-J
Copy link

Hoper-J commented Mar 15, 2024

This solved my issue😄. You should replace port with your proxy server's port number:

import os

os.environ['http_proxy'] = 'http://127.0.0.1:port'
os.environ['https_proxy'] = 'http://127.0.0.1:port'
os.environ['all_proxy'] = 'socks5://127.0.0.1:port'

@observerkei
Copy link

我也遇到了这个问题 GRPC 返回 503,但我的错误是 google.api_core.exceptions.RetryError: Deadline of 60.0s超过调用目标函数时,最后一个异常:503 无法连接到所有地址;最后一个错误:未知:ipv4:172.217.160.74:443:无法连接到远程主机:FD 关闭。 IP无法访问

You can bypass GRPC and use rest:

genai.configure(api_key=GOOGLE_API_KEY,transport='rest')

Thanks for sharing! The problem that has bothered me for a long time does not seem to be found in the official document 😂

@gyp3085000
Copy link

genai.configure(api_key=GOOGLE_API_KEY,transport='rest')
The problem can be solved, but the streaming Q&A output will fail.

@MizukiRin
Copy link

我也遇到了这个问题 GRPC 返回 503,但我的错误是 google.api_core.exceptions.RetryError: Deadline of 60.0s超过调用目标函数时,最后一个异常:503 无法连接到所有地址;最后一个错误:未知:ipv4:172.217.160.74:443:无法连接到远程主机:FD 关闭。 IP无法访问

You can bypass GRPC and use rest:

genai.configure(api_key=GOOGLE_API_KEY,transport='rest')

Thanks! It really works.

@markmcd
Copy link
Member

markmcd commented Aug 27, 2024

Lots happened in this thread - is there anything left we can help with? If not, we'll close this one.

@markmcd markmcd added the status:awaiting user response Awaiting a response from the author label Aug 27, 2024
Copy link

Marking this issue as stale since it has been open for 14 days with no activity. This issue will be closed if no further activity occurs.

@github-actions github-actions bot added the status:stale Issue/PR will be closed automatically if there's no further activity label Sep 11, 2024
Copy link

This issue was closed because it has been inactive for 28 days. Please post a new issue if you need further assistance. Thanks!

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Sep 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component:python sdk Issue/PR related to Python SDK status:awaiting user response Awaiting a response from the author status:stale Issue/PR will be closed automatically if there's no further activity type:bug Something isn't working
Projects
None yet
Development

No branches or pull requests