[Bug] Claude 3.5 在图片的内容分析上有问题

[breakstring]

📦 Environment

Vercel

📌 Version

1.1.6

💻 Operating System

macOS

🌐 Browser

Other

🐛 Bug Description

无法使用 Claude 3.5 解析图片内容，OpenAI 则没问题。

📷 Recurrence Steps

1. 选择模型为 Claude 3.5，插入图片到聊天窗口中，输入 prompt 请求解析。
2. 报错提示如下：
   、、、
   {
   "error": {
   "error": {
   "type": "error",
   "error": {
   "type": "invalid_request_error",
   "message": "messages.0.content.1.image.source.base64.media_type: Input should be 'image/jpeg', 'image/png', 'image/gif' or 'image/webp'"
   }
   },
   "status": 400,
   "headers": {
   "via": "1.1 google",
   "date": "Sun, 23 Jun 2024 23:28:03 GMT",
   "connection": "keep-alive",
   "request-id": "req_01BhNKq3EqhyioScf16uDPm5",
   "content-type": "application/json",
   "content-length": "193",
   "x-should-retry": "false",
   "x-cloud-trace-context": "b8d7344e29c2adb51dfd7c01eb5a9e9f"
   }
   },
   "endpoint": "https://api.anthropic.com",
   "provider": "anthropic"
   }
   、、、
   

🚦 Expected Behavior

2. 返回解析的内容

📝 Additional Information

No response

[lobehubbot]

👀 @breakstring

Thank you for raising an issue. We will investigate into the matter and get back to you as soon as possible.
Please make sure you have given us as much context as possible.
非常感谢您提交 issue。我们会尽快调查此事，并尽快回复您。 请确保您已经提供了尽可能多的背景信息。

[arvinxx]

看报错应该是 Anthropic 的api不支持base64 的图片，必须要可访问的 url 才能识别。你配置下s3应该就好了

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

The error message seems to be that Anthropic's API does not support base64 images, and an accessible URL is required to identify them. It should be fine if you configure s3

[breakstring]

看报错应该是 Anthropic 的api不支持base64 的图片，必须要可访问的 url 才能识别。你配置下s3应该就好了

我是采用的服务端数据库存储方案，目前是华为云兼容 S3 的 OBS。
BTW: 请求 OpenAI 正常。

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Judging from the error report, it should be that Anthropic's API does not support base64 images, and an accessible URL is required to identify them. It should be fine if you configure s3

The server-side database storage solution I use is currently OBS compatible with Huawei Cloud's S3.
BTW: Requesting OpenAI works fine.

[arvinxx]

那估计是个 bug，我晚点看看

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

That's probably a bug, I'll check it out later

[breakstring]

那估计是个 bug，我晚点看看

Any update?

[arvinxx]

https://docs.anthropic.com/en/docs/build-with-claude/vision#can-claude-read-image-urls 查阅了下官方文档， Claude API 目前不支持传入 url 来识别，如果要使用必须要传输 base64。这么看和 Google 的 Gemini 问题一样，可能还是得用 base64 传输

[hominsu]

any update?

[huangyafei]

确实是个影响挺大的问题，请问有什么进展了么？

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

It is indeed an issue with a huge impact. Is there any progress?

[arvinxx]

Here is the PR to fix similar issue: #3370

[wxy-skye]

The problem still persists. The server-side mode cannot use Claude for image recognition. It can only use GPT.

[breakstring]

The problem still persists. The server-side mode cannot use Claude for image recognition. It can only use GPT.
Yes, the PR #3370 still in progress...

[hlc1209]

+1，严重影响日常使用

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

+1, seriously affects daily use

[adminewacc]

me too, pls fix that

[RetrainIt]

+1，严重影响日常使用

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

+1, seriously affects daily use

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

I have received your letter, thank you.

[2019YKL]

S3有办法本地的那个S3么，我识图目前还没成功过（

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Is there any way to localize S3 to S3? I haven’t been able to successfully read images yet (

[Stardust-SJF]

I wonder if it is possible to use base64 to directly upload images to the model provider instead of uploading images into the knowledge database, or convert the image to base64 after retrieving it from S3 storage and then encode (to base 64) and send them to model providers?

[adminewacc]

I wonder if it is possible to use base64 to directly upload images to the model provider instead of uploading images into the knowledge database, or convert the image to base64 after retrieving it from S3 storage and then encode (to base 64) and send them to model providers?

i solved it

[adminewacc]

S3有办法本地的那个S3么，我识图目前还没成功过（

in this case i think that it is showing this error because of your api key quota

[lobehubbot]

✅ @breakstring

This issue is closed, If you have any questions, you can comment and reply.
此问题已经关闭。如果您有任何问题，可以留言并回复。

[lobehubbot]

🎉 This issue has been resolved in version 1.15.30 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀