Can't get local llama3 to answer notes-related questions

[Brenzle]

I installed llama 3 8b with Ollama, and when my PC starts Ollama is running (small icon in taskbar). I can access it through CMD (I'm on Windows 10) and it works as expected (answers regular questions).

In Obsidian, I got the SC plugin version 2.1.66 installed and in the SC conversation, I set the model name as in the picture and an API key that I got from Open Router. Once I save, there's a popup saying "No Smart Connections found" and another one saying "Success! API key is valid.". Regular questions work but if I ask for a note summary, the reply looks like in the picture. If I ask a "Based on my notes" question, it returns "context: <note_name1>, <note_name2>..." and keeps loading forever.

I'm not very familliar with this and it seems to me like the readme file assumes some prior knowledge.

What am I doing wrong? I'd appreciate it if someone could point it out to me, and/or give me some steps to get llama to work with my notes in the SC conversation.

[brianpetro]

Hi @Brenzle

I moved this to the discussion sections because it isn't a particular issue with Smart Connections that needs to be solved.

My understanding is that you're trying to utilize a local model.

However, the screenshot indicates the Smart Chat is configured to use Open Router, a cloud service that runs open-source and other models.

Running a local model does indeed require background knowledge about using Ollama or similar software, and how to do that is a bit outside the scope of Smart Connections.

While I can't help much further than this, here is a screenshot of what configuring the Smart Chat to use a local model might look like:

Hopefully, someone else from the community, someone with more experience implementing Ollama, will be able to help you further 😊

🌴

[Brenzle]

Thanks for creating the discussion, indeed makes much more sense. Thanks for the info too, still hoping someone can help.

[SyndicatedPillbug]

I'd love some instructions on how to implement ollama as well. I can download/chat/use it from terminal, I'm just not certain how to connect it to Smart Chat

[coulas]

I tried /api/chat (Ollama's API) and /v1/api.chat (Ollama support OpenAI's API) and none works in either Custom Local nor Custom API.
Using /v1, there is an API error asking for "console logs", using Ollama it starts a model (send question, run "ollama ps" in a shell and the select model is running but it will never answer even "Ollama ps" shows the model has shutdown

[princinv]

glad i found this thread, was about to post about the same thing. the api requirement seems to be the issue.

@brianpetro id be happy to toss together a first draft of a guide for the docs when this gets sorted out.

[brianpetro]

Yeah, that would be awesome.

For future reference, contributing to the docs can be done here https://github.com/brianpetro/Smart-Connections-Docs
🌴

[DataStrategist]

Greetings all! New to the party, I think I can help here. See the screenshot:

1. Ollama's endpoint (as of right now) is definitely api/generate (you can see the curl command worked w/ that endpoint)
2. @brianpetro it appears that your plugin is looking for an API key even for local models? That's prolly a bug, no?
   
3. Also for @brianpetro , I think you're having a problem parsing the output of the ollama model, probably on get_text_chunk_from_stream ? (as u can see in the developer screen error). Does that help? I tried turning streaming off, but that didn't work.

I'm on the latest model of your plugin

[brianpetro]

Thanks for letting me know.

Try turning off the "streaming" setting to see if that works 🌴

[DataStrategist]

I mentioned in point 3. that I tried turning the streaming off. Broke everything :)

[brianpetro]

Whoops. I'll have to do some testing to see what's up 🌴

[DataStrategist]

@princinv if you're writing a guide, pls see here... there's several things here that we know DO work... just have to get those last few purring :)