Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to increase speech to text speed when using whisper cpp? #1635

Open
ITHealer opened this issue Dec 14, 2023 · 5 comments
Open

How to increase speech to text speed when using whisper cpp? #1635

ITHealer opened this issue Dec 14, 2023 · 5 comments

Comments

@ITHealer
Copy link

Currently I am using whisper_tiny.en.tflite for my android project. And the inference time for speech to text is quite long.
https://github.com/usefulsensors/openai-whisper/tree/main/android_app/Whisper-TFLIte-Android-Example


I/flutter ( 3449): Model name: /assets/whisper-tiny.en.tflite
I/flutter ( 3449): Load success
I/flutter ( 3449): Load model time 0:00:03.274847
I/htotext_exampl( 3449): ProcessProfilingInfo new_methods=1340 is saved saved_to_disk=1 resolve_classes_delay=8000
I/tflite ( 3449): Initialized TensorFlow Lite runtime.
W/1.ui ( 3449): type=1400 audit(0.0:15783): avc: denied { read } for name="u:object_r:vendor_default_prop:s0" dev="tmpfs" ino=14533 scontext=u:r:untrusted_app:s0:c127,c257,c512,c768 tcontext=u:object_r:vendor_default_prop:s0 tclass=file permissive=0
E/libc ( 3449): Access denied finding property "ro.hardware.chipname"
I/tflite ( 3449): Created TensorFlow Lite XNNPACK delegate for CPU.
I/flutter ( 3449): Result process audio file: I love you.
I/flutter ( 3449): Processing time 0:00:11.894501


Is there any way to shorten the time, please help me?
Thanks!

@ITHealer
Copy link
Author

@ggerganov Can you give me directions?

@dgm3333
Copy link

dgm3333 commented Dec 14, 2023

buy a more powerful processor?
Given that you haven't even given basic details of your setup it seems to me you are being pretty unreasonable to target the primary dev for something which is always going to be limited by hardware and is a totally different repo to this one :-(

@themanyone
Copy link

themanyone commented Dec 19, 2023

I am the author of Caption Anything and Whisper Dictation. In #1653, I mentioned how speed can be improved by compiling Whisper.cpp with acceleration like CLBlast or cuBLAS, using the tiny models, or employing a client-server setup. Connect slow clients like android to faster computers running Whisper.cpp ./server instances on the network.

I also ran into a case where generation was inexplicably slow, and it was no longer using the GPU. The solution was to reload the video driver module (or reboot). (I was testing -allow-unsupported-compiler thinking I could outsmart the system).

sudo modprobe -r nvidia_uvm; sudo modprobe nvidia_uvm

@jordibruin
Copy link

Android phones won't be able to run this fast.

@zhouwg
Copy link
Contributor

zhouwg commented Mar 4, 2024

Xiaomi 14 could do that(with Xiaomi's proprietary 6B device-side AI model) because it contains a very powerful mobile SoC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants