Sometimes the LLM will just not generate a response randomly.

[WillMiles]

As title. I'm using LLMCharacter.Complete to generate responses. I have debug mode on and it shows: https://pastebin.com/E6jSGmUC
Which is normal until the last two lines. Here's compared to when it properly finishes the completion: https://pastebin.com/YNwreizL
The incorrect one is missing the lines stating that the slot is processing a task, so that seems like the issue but it isn't clear why that is happening to me.

The completion just suspends forever, it doesn't eventually finish or anything like that. Though I can continue to generate responses afterwards if I restart it after a few seconds. I suspect an error has been thrown in LlamaFile but I'm not sure how I would detect or debug that.

The prompt is the exact same in both situations. It may have something to do with simultaneous prompts. But sometimes simultaneous prompts work perfectly fine. I've also tried to wait for a response to finish generating before the next one starts but that just makes the game wait forever because the response never comes. This happens regardless of other settings like which model or what the prompt is. But for reference I'm using Qwen2-1.5B-Q8.0 for this test. (Which is giving great results other than this!) If anyone has any experience with this issue I would love to hear it, I've done a lot of testing and can't seem to pin it down.

Thank you!

[amakropoulos]

hi there 🙂! I would need some more info.
Are you using a sample?
How does your server command look like in the debug messages?
Could you send a screenshot of the LLM and LLMCharacter GameObjects and (part of) the code you are using?

[WillMiles]

Not using a sample. The code that interacts with your system with just this line:

    public void GenerateSentence(string prompt, AIManager.BotSentenceOptions options, LLMCharacter llmCharacter, Action<string> onComplete, List<string> stopwords = null)
    {
        _ = llmCharacter.Complete(prompt, (str) => onComplete(str));
    }

Everything runs through that function. The BotSentenceOptions and stopwords aren't even used here, as you can see.


(I have no idea if that's how the penalty prompt is supposed to work I couldn't find any documentation about that. Regardless it still breaks without that set.)

What do you mean by how does the server command look? The only debug messages I see are those. Do I not have a setting set?

Thanks for the consideration and let me know if you need to see anything else!

[WillMiles]

I do programmatically change the temperature when I try to generate a new response, I'm not sure if that's relevant though.

[amakropoulos]

I don't see something off 🤔.
Could you try with a larger Num Predict and/or the Q4_K_M model instead of Q8?

[amakropoulos]

which LLMUnity version are you using? LlamaFile is not used since v2.x.
You can check your Editor.log to see if it shows something.

[WillMiles]

Oh my bad, I just saw that you mentioned it earlier. I'm using the newest version, so I guess no LlamaFile. I'll check out the Editor.log.

[amakropoulos]

No worries I just wanted to figure out if you are on an old one 🙂.
I will try your settings as well to see if I can reproduce it.

[WillMiles]

This is what the Editor.Log says, pretty much the same story. It does seem to occur more often when using more complicated prompts though that's more of a vibe than anything.
https://pastebin.com/z7ydGe64

[WillMiles]

Ok so after some more testing I've got this log: https://pastebin.com/V6ztFmSS
It says "generation eval time = 0.01 ms" which sounds like a problem. If I keep retrying with the same prompt over and over again it doesn't work. It's so weird because if I run a different prompt later then it inexplicably does work somehow? For that one particular prompt at that time it will just never generate a sentence (even though literally the exact same prompt was working and generating just a moment before). I even did testing where it is definitely the only prompt running at that time and it still doesn't work sometimes.

[WillMiles]

I'm very sorry, the problem was with my code. It was throwing an error when editing the sentence, but it's weird that it wasn't anywhere on the logs which is why I thought that it must be the LLM. I wonder if it has something to do with how the onComplete method is invoked but I'm not sure.
Thanks for the guidance, you definitely pointed me in the right direction.

[amakropoulos]

Great! Yes, Unity is really bad to debug 🙂.
Let me know if you get stuck with anything else!