-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added whisper state + default state on the whisper_context #523
Added whisper state + default state on the whisper_context #523
Conversation
Hey @ggerganov, Let me know what you think about this simpler approach. If you'll like it, we can create some follow-up tasks to update emscripten + various samples to not create so many contexts anymore and save some memory. :) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice - well done!
We need to update the callbacks in the examples.
If you'll like it, we can create some follow-up tasks to update emscripten + various samples to not create so many contexts anymore and save some memory. :)
Hmm, where do we create many contexts?
Here for example: https://github.com/ggerganov/whisper.cpp/blob/master/examples/stream.wasm/emscripten.cpp#L16 But there are other samples which are creating multiple contexts like that. |
@sandrohanea |
…gerganov#523) * Added whisper state + default state on the whisper_context * Fixed some examples and bindings * Fixed whisper_n_len (which was used in some binding) and added whisper_n_len_from_state * Fixed comments * whisper : reuse kv_cache_free() and fix compiler warnings * whisper : clean-up the API comments --------- Co-authored-by: Sandro Hanea <sandrohanea@microsoft.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
…gerganov#523) * Added whisper state + default state on the whisper_context * Fixed some examples and bindings * Fixed whisper_n_len (which was used in some binding) and added whisper_n_len_from_state * Fixed comments * whisper : reuse kv_cache_free() and fix compiler warnings * whisper : clean-up the API comments --------- Co-authored-by: Sandro Hanea <sandrohanea@microsoft.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
…gerganov#523) * Added whisper state + default state on the whisper_context * Fixed some examples and bindings * Fixed whisper_n_len (which was used in some binding) and added whisper_n_len_from_state * Fixed comments * whisper : reuse kv_cache_free() and fix compiler warnings * whisper : clean-up the API comments --------- Co-authored-by: Sandro Hanea <sandrohanea@microsoft.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
…gerganov#523) * Added whisper state + default state on the whisper_context * Fixed some examples and bindings * Fixed whisper_n_len (which was used in some binding) and added whisper_n_len_from_state * Fixed comments * whisper : reuse kv_cache_free() and fix compiler warnings * whisper : clean-up the API comments --------- Co-authored-by: Sandro Hanea <sandrohanea@microsoft.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
…gerganov#523) * Added whisper state + default state on the whisper_context * Fixed some examples and bindings * Fixed whisper_n_len (which was used in some binding) and added whisper_n_len_from_state * Fixed comments * whisper : reuse kv_cache_free() and fix compiler warnings * whisper : clean-up the API comments --------- Co-authored-by: Sandro Hanea <sandrohanea@microsoft.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
In order to reuse memory and run multiple transformations on a single model, added the whisper_state (which is keeping the state of the current transformation).
In order to not be a breaking change and keep the usage of whisper_context to minimum, I added a default_state on the context and the current methods (e.g. whisper_full) will use that context in order to run the transformation.
Ofc, these methods are not thread safe, but additional methdos are created where the state can be externally provided (e.g.
whisper_full_with_state
)For the callbacks, the state is also provided so bindings need to be updated for this, but the other methods are still valid.