-
Notifications
You must be signed in to change notification settings - Fork 612
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: running local models #269
Conversation
works beautifully :) To test it, I deployed something with a wrong tag, so it was failing to pull the image and bring up the service, that was the response (it took some time, but hey, that's now all local!):
My setup:
To bring up the API server (locally):
Will polish, and add docs alongside with the PR, stay tuned! |
here looks good! - let me know if that's the good direction @AlexsJones , I've also updated the docs, feedback is welcome :) |
Looks very exciting @mudler , @matthisholleville @arbreezy perhaps we use this as another reason to have a configuration Object for AI rather than passing more strings. Also @mudler I think having your own AI struct might be easier ( yes its a little more work, but it eventually means less manipulation of the openai.go) |
Thanks for the feedback :) Going to work on your feedback into separate commits and be back at you as soon as possible! |
@arbreezy @AlexsJones do you want me to keep the |
I think we can leave it in there unused, as you say it might be useful eventually! |
25a00e7
to
d1c32f8
Compare
@AlexsJones @arbreezy implemented your suggestions - added some comments inline with the changes introduced |
This is how it works now:
|
Very useful feature ! Good job @mudler !! |
f6ec65c
to
dbb52d0
Compare
Rebased from main, docs updated, and gave it another shot locally!
|
There was a small bug introduced in the #310 PR, sorry about that, I think it was an oversight between cherrypicks and rebases here. I've added a fix for it in this PR. (see my comment above) @AlexsJones @arbreezy @matthisholleville rebased, re-tested locally, good to go here! please let me know if anything is missing! |
Signed-off-by: mudler <mudler@mocaccino.org>
Signed-off-by: mudler <mudler@mocaccino.org>
I'll test this tonight! Thank you for your contribution |
I am back tomorrow, I will have a look as well. |
Hi, I just tested it and here's the error I encountered: ➜ k8sgpt git:(9b914fb) ./k8sgpt analyze --explain --backend localai --namespace k8sgpt --no-cache
Warning: Legacy config file at `/Users/matthisholleville/.k8sgpt.yaml` detected! This file will be ignored!
0% | | (0/1, 0 it/hr) [0s:0s]
Error: failed while calling AI provider localai: error, json: cannot unmarshal string into Go struct field ErrorResponse.error of type map[string]json.RawMessage Here's my setup:
|
Good catch! that was a miss from my side, I just traced it back, now the API returns OpenAI error types: mudler/LocalAI#80
This model is for "whisper" - it is not for LLM/GPTs. I've updated the docs with an e2e example for gpt4all-j: https://github.com/go-skynet/LocalAI#example-use-gpt4all-j-model
My bad, docs weren't aligned, you need to turn debug mode (by either setting the DEBUG env var to true, or specifying it in the CLI). I've updated the docs as well! (https://github.com/go-skynet/LocalAI#api) |
Signed-off-by: mudler <mudler@mocaccino.org>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK LGTM,
it's working great with a local model
I suggest to merge the localai.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Hey, @mudler , Can you contribute a step by step doc guide to download a pre-trained model from somewhere , and use your feature ? |
This is an untested draft stillIt works! see my comment below. This should be enough for testing on an OpenAI compatible endpoint just by letting the user change the base_url, so should work as well with https://github.com/go-skynet/LocalAII'll experiment a bit locally and refine this, adding docs too. cc: @arbreezy @AlexsJones
Closes #188
📑 Description
✅ Checks
ℹ Additional Information