how to get unbuffered responses?

[alph4b3th]

I noticed that .predict returns a complete string, which is the model's response. However, I need to give the user a feeling of iteration in which the model must send what it is "typing" in time to the user's client. But the .predict function only returns post-finished responses. How can I get an answer as predicted by the model? A feeling of "typing"

[mudler]

This is not implemented yet - however it's something I'm interested in supporting.

[mudler]

one way would be to restrict the binding to send back data directly to a golang channel, for instance: https://github.com/matiasinsaurralde/cgo-channels/tree/master however, I see still that could incur in a huge penalty, as context switch from golang and C in a loop have a high computational cost.

I think we could offer a low-level functionality to address the specific case and scope it to have just a few functions exposed, but wouldn't suggest usage when performance is a requirement.

[alph4b3th]

one way would be to restrict the binding to send back data directly to a golang channel, for instance: https://github.com/matiasinsaurralde/cgo-channels/tree/master however, I see still that could incur in a huge penalty, as context switch from golang and C in a loop have a high computational cost.

I think we could offer a low-level functionality to address the specific case and scope it to have just a few functions exposed, but wouldn't suggest usage when performance is a requirement.

can you create this functionality?

[noxer]

To answer the original question:

llama.SetTokenCallback(func(token string) bool {
    fmt.Print(token)
    return true // we want the predictor to continue
})

[mudler]

I think we can close this now, thanks @noxer ❤️!