-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to increase speech to text speed when using whisper cpp? #1635
Comments
@ggerganov Can you give me directions? |
buy a more powerful processor? |
I am the author of Caption Anything and Whisper Dictation. In #1653, I mentioned how speed can be improved by compiling Whisper.cpp with acceleration like CLBlast or cuBLAS, using the tiny models, or employing a client-server setup. Connect slow clients like android to faster computers running Whisper.cpp I also ran into a case where generation was inexplicably slow, and it was no longer using the GPU. The solution was to reload the video driver module (or reboot). (I was testing -allow-unsupported-compiler thinking I could outsmart the system).
|
Android phones won't be able to run this fast. |
Xiaomi 14 could do that(with Xiaomi's proprietary 6B device-side AI model) because it contains a very powerful mobile SoC |
Currently I am using whisper_tiny.en.tflite for my android project. And the inference time for speech to text is quite long.
https://github.com/usefulsensors/openai-whisper/tree/main/android_app/Whisper-TFLIte-Android-Example
I/flutter ( 3449): Model name: /assets/whisper-tiny.en.tflite
I/flutter ( 3449): Load success
I/flutter ( 3449): Load model time 0:00:03.274847
I/htotext_exampl( 3449): ProcessProfilingInfo new_methods=1340 is saved saved_to_disk=1 resolve_classes_delay=8000
I/tflite ( 3449): Initialized TensorFlow Lite runtime.
W/1.ui ( 3449): type=1400 audit(0.0:15783): avc: denied { read } for name="u:object_r:vendor_default_prop:s0" dev="tmpfs" ino=14533 scontext=u:r:untrusted_app:s0:c127,c257,c512,c768 tcontext=u:object_r:vendor_default_prop:s0 tclass=file permissive=0
E/libc ( 3449): Access denied finding property "ro.hardware.chipname"
I/tflite ( 3449): Created TensorFlow Lite XNNPACK delegate for CPU.
I/flutter ( 3449): Result process audio file: I love you.
I/flutter ( 3449): Processing time 0:00:11.894501
Is there any way to shorten the time, please help me?
Thanks!
The text was updated successfully, but these errors were encountered: