[Request] LLM 适配

[arvinxx]

独立 LLM 设置 Tab，为上层助手角色提供丰富的大模型选项。

清单：

• Azure OpenAI [Request] 支持 Azure OpenAI #131
• Cluade [Request] 是否可以支持Claude #83
• Replacaite LLM https://replicate.com/
• 本地 LLM - LocalAI
  • LLaMA
  • Vinuna
  • ChatGLM 6B
  • Qwen-7B
  • ToolLLaMA

完整 RFC 在： #737

[ddwinhzy]

Would like to add OpenAI's fine-tuned model call

[arvinxx]

Would like to add OpenAI's fine-tuned model call

Yes, with custom model name

[gavinliu]

本地 LLM 是否有一个明确的支持计划？

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Does the local LLM have a clear support plan?

[jyboy]

Google's Gemini Pro API is also available now.
Please see the official tweet.

[arvinxx]

本地 LLM 是否有一个明确的支持计划？

计划在1月份开始做

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Does the local LLM have a clear support plan?

Plan to start in January

[hazelnutcloud]

any plans to add support for mistral?

[arvinxx]

yes

[structure-charger]

推荐 https://github.com/xorbitsai/inference

[WayneShao]

我是用了 2023-12-01-preview 版本的API调用 Azure 的 gpt-4-vision 模型，可以成功获得回复，但是回复一句之后就会直接截断

控制台输出

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

I used the 2023-12-01-preview version of the API to call Azure's gpt-4-vision model. I could successfully get a reply, but the reply would be truncated directly after one sentence.

console output

[arvinxx]

我是用了 2023-12-01-preview 版本的API调用 Azure 的 gpt-4-vision 模型，可以成功获得回复，但是回复一句之后就会直接截断

控制台输出

这个应该是 gpt-4-vision 内置的max_tokens 太小了导致的

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

I used the 2023-12-01-preview version of the API to call Azure's gpt-4-vision model. I could successfully get a reply, but the reply would be truncated directly after one sentence.

Console output

This should be caused by the built-in max_tokens of gpt-4-vision being too small.

[Met-Du]

Are there any plans for Gemini Pro? Thank you

[cocobeach]

独立 LLM 设置 Tab，为上层助手角色提供丰富的大模型选项。

清单：

• Azure OpenAI [Request] 支持 Azure OpenAI #131

• Cluade [Request] 是否可以支持Claude #83

• Replacaite LLM https://replicate.com/

• 本地 LLM - LocalAI

  • LLaMA
  • Vinuna
  • ChatGLM 6B
  • Qwen-7B
  • ToolLLaMA
  • mistral

完整 RFC 在： #737

Hi Team!
There is a very simple simple way of running models locally with ollama https://github.com/ollama/ollama and wrap it up with https://github.com/BerriAI/litellm and using https://github.com/BerriAI/liteLLM-proxy it shouldn't be too difficult to make it work with a lot of OS LLM.
Best Regards

[cocobeach]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

Are there any plans to adapt to ollama?

ollama wrapped in litellm should be a piece of cake, I am going to try it.

[arvinxx]

@cocobeach Thanks for your introduction！ We will add ollama support also~ 😁

[j0ole]

Are thers any plants to add a key pool and calling weight to dynamic call those keys?

[arvinxx]

Are thers any plants to add a key pool and calling weight to dynamic call those keys?

No. There is not plan about key polling.

[zhangheli]

有没有可能，直接接入 langchain 内 chat model 的支持 ？

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Is it possible to directly access the chat model support in langchain?

[arvinxx]

@zhangheli 试过了，不行：#737 (reply in thread)

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

@zhangheli tried it, but it didn’t work: #737 (reply in thread)

[cocobeach]

@cocobeach Thanks for your introduction！ We will add ollama support also~ 😁

That's great, it would allow Mac users to use the interface with local models easily.

Also for the best inference but that's for nvidia therefore mostly windows station vLLM is quite unique in the way it manages gpu memory, for instance for me it's the only setup that allows me to actually take advantage of my dual RTX a4000 setup but I can run Mixtral on it quite smoothly and with excellent results.
Now that gpt4 got "lazy" I think there is a good opening in the market for open source models, if they are managed properly.

I really like your platform, but I don't understand why the total amount of tokens allowed for a whole chat thread is limited to the model's context windows. Meaning if I am using gpt 3.5 turbo 16k, I get to chat up to 16k thread size, then it gives me an error?
Can I configure that differently?
Best Regards

[arvinxx]

@cocobeach you can set a limit of current topic context. We don't set a limit by default.

[cocobeach]

Tanks a lot!!

[ifsheldon]

Please add Moonshot support! See https://platform.moonshot.cn
The APIs are a compatible subset with OpenAI's

[sjy]

@cocobeach Thanks for your introduction！ We will add ollama support also~ 😁

This a PR WIP for Ollama Provider, #1265 @arvinxx

[lobehubbot]

✅ @arvinxx

This issue is closed, If you have any questions, you can comment and reply.
此问题已经关闭。如果您有任何问题，可以留言并回复。

[lobehubbot]

🎉 This issue has been resolved in version 0.127.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀