Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Cross-Region Inference for Bedrock Chat Models #1113

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

srdas
Copy link
Collaborator

@srdas srdas commented Nov 20, 2024

Fixes Issue #1114

Fixes PR #1014
(supersedes this PR, which is being closed)

Cross region inference is now supported in Bedrock. In particular this is available for the latest models such as Llama3.2 and Claude-Sonnet-3.5, etc.

This PR adds all Llama3.2 models to the list of available models.

Also added is a cross region inference area field required if a model is being used from another region, which is possible now for models not available in the user's region.

If a model is not available in a region, then using it will generate an error:
image

File "/opt/anaconda3/envs/jupyter-ai/lib/python3.12/site-packages/langchain_aws/llms/bedrock.py", line 857, in _prepare_input_and_invoke_stream
    raise ValueError(f"Error raised by bedrock service: {e}")
ValueError: Error raised by bedrock service: An error occurred (ValidationException) when calling the InvokeModelWithResponseStream operation: Invocation of model ID meta.llama3-2-3b-instruct-v1:0 with on-demand throughput isn’t supported. Retry your request with the ID or ARN of an inference profile that contains this model.

To avail of cross region inference, enter the area from this page into the field provided:
image
The model will now be available and will not error out:
image

@srdas srdas added the enhancement New feature or request label Nov 20, 2024
@srdas srdas assigned srdas and dlqqq Nov 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants