Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Bedrock cross region inference #8190

Closed
4 of 5 tasks
SIN1109 opened this issue Sep 10, 2024 · 1 comment · Fixed by #8402
Closed
4 of 5 tasks

Support Bedrock cross region inference #8190

SIN1109 opened this issue Sep 10, 2024 · 1 comment · Fixed by #8402

Comments

@SIN1109
Copy link

SIN1109 commented Sep 10, 2024

Self Checks

  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue,否则会被关闭。谢谢!:)
  • Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

Can you consider to support AWS bedrock cross region inference, like Claude 3.5 sonnet model
https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference.html

2. Additional context or comments

https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference.html

3. Can you help us with this feature?

  • I am interested in contributing to this feature.
Copy link

dosubot bot commented Sep 10, 2024

Dify currently supports AWS Bedrock as one of the Model-as-a-Service (MaaS) vendors, including the ability to specify the AWS region for model inference. However, there is no specific documentation or code available in the repository that outlines how to set up cross-region inference for AWS Bedrock models.

To support cross-region inference, you would need to configure the region_name in the Config object when setting up the Bedrock runtime client. Here is an example of how this can be done:

from botocore.config import Config
import boto3

client_config = Config(
    region_name='us-west-2'  # Specify the desired AWS region
)

bedrock_runtime = boto3.client(
    service_name='bedrock-runtime',
    config=client_config,
    aws_access_key_id='your_access_key',
    aws_secret_access_key='your_secret_access_key'
)

This configuration ensures that the Bedrock runtime client operates in the specified AWS region, which is crucial for cross-region inference [1].

For more detailed steps on deploying and configuring cross-region inference, you can refer to the AWS Bedrock documentation.

To continue talking to Dosu, mention @dosu.

ybalbert001 pushed a commit to ybalbert001/dify that referenced this issue Sep 12, 2024
ybalbert001 pushed a commit to ybalbert001/dify that referenced this issue Sep 14, 2024
crazywoola pushed a commit that referenced this issue Sep 14, 2024
…to distinguish between different region groups) (#8402)

Co-authored-by: Yuanbo Li <ybalbert@amazon.com>
ZhouhaoJiang added a commit that referenced this issue Sep 14, 2024
* main: (40 commits)
  feat: allow users to specify timeout for text generations and workflows by environment variable (#8395)
  Fix: operation postion of answer in logs (#8411)
  fix: when the variable does not exist, an error should be prompted (#8413)
  fix(workflow): the answer node after the iteration node containing the answer was output prematurely (#8419)
  fix:logs and rm unused codes in CacheEmbedding (#8409)
  fix: resolve runtime error when self.folder is None (#8401)
  Fix: Support Bedrock cross region inference #8190 (Update Model name to distinguish between different region groups) (#8402)
  fix(docker): aliyun oss path env key (#8394)
  fix: pyproject.toml typo (#8396)
  fix: o1-mini 65563 -> 65536 (#8388)
  fix: sandbox issue related httpx and requests (#8397)
  chore: improve usage of striping prefix or suffix of string with Ruff 0.6.5 (#8392)
  fix (#8322 followup): resolve the violation of pylint rules (#8391)
  chore: refurish python code by applying Pylint linter rules (#8322)
  support hunyuan-turbo (#8372)
  chore: update firecrawl scrape to V1 api (#8367)
  fix(workflow): both parallel and single branch errors occur in if-else (#8378)
  fix: edit load balancing not pass id (#8370)
  fix: add before send to remove langfuse defaultErrorResponse (#8361)
  fix: when edit load balancing config not pass the empty filed value hidden (#8366)
  ...
ybalbert001 pushed a commit to ybalbert001/dify that referenced this issue Sep 14, 2024
crazywoola pushed a commit that referenced this issue Sep 14, 2024
Co-authored-by: Yuanbo Li <ybalbert@amazon.com>
ZhouhaoJiang added a commit that referenced this issue Sep 14, 2024
* feat/new-login: (27 commits)
  chore: format
  feat: add fulfill_login_params_from_env
  feat: update message
  feat: update mismatch description
  feat: remove env example
  chore: PLR6201 Use a set literal when testing for membership
  feat: update EmailOrPasswordMismatchError
  feat: allow users to specify timeout for text generations and workflows by environment variable (#8395)
  Fix: operation postion of answer in logs (#8411)
  fix: when the variable does not exist, an error should be prompted (#8413)
  fix(workflow): the answer node after the iteration node containing the answer was output prematurely (#8419)
  fix:logs and rm unused codes in CacheEmbedding (#8409)
  fix: resolve runtime error when self.folder is None (#8401)
  Fix: Support Bedrock cross region inference #8190 (Update Model name to distinguish between different region groups) (#8402)
  fix(docker): aliyun oss path env key (#8394)
  fix: pyproject.toml typo (#8396)
  fix: o1-mini 65563 -> 65536 (#8388)
  fix: sandbox issue related httpx and requests (#8397)
  chore: improve usage of striping prefix or suffix of string with Ruff 0.6.5 (#8392)
  fix (#8322 followup): resolve the violation of pylint rules (#8391)
  ...
LLmoskk pushed a commit to LLmoskk/dify that referenced this issue Sep 16, 2024
Co-authored-by: Yuanbo Li <ybalbert@amazon.com>
hos715 pushed a commit to hos715/dify that referenced this issue Sep 16, 2024
Co-authored-by: Yuanbo Li <ybalbert@amazon.com>
hos715 pushed a commit to hos715/dify that referenced this issue Sep 16, 2024
Co-authored-by: Yuanbo Li <ybalbert@amazon.com>
laipz8200 pushed a commit that referenced this issue Sep 18, 2024
Co-authored-by: Yuanbo Li <ybalbert@amazon.com>
cuiks pushed a commit to cuiks/dify that referenced this issue Sep 26, 2024
…odel name to distinguish between different region groups) (langgenius#8402)

Co-authored-by: Yuanbo Li <ybalbert@amazon.com>
cuiks pushed a commit to cuiks/dify that referenced this issue Sep 26, 2024
Co-authored-by: Yuanbo Li <ybalbert@amazon.com>
lau-td pushed a commit to heydevs-io/dify that referenced this issue Oct 23, 2024
…odel name to distinguish between different region groups) (langgenius#8402)

Co-authored-by: Yuanbo Li <ybalbert@amazon.com>
lau-td pushed a commit to heydevs-io/dify that referenced this issue Oct 23, 2024
Co-authored-by: Yuanbo Li <ybalbert@amazon.com>
idonotknow pushed a commit to AceDataCloud/Dify that referenced this issue Nov 16, 2024
…odel name to distinguish between different region groups) (langgenius#8402)

Co-authored-by: Yuanbo Li <ybalbert@amazon.com>
idonotknow pushed a commit to AceDataCloud/Dify that referenced this issue Nov 16, 2024
Co-authored-by: Yuanbo Li <ybalbert@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants