Android Inference is too slow

[saffie91]

Hi, trying out the Android Implementation, it works well overall and thank you for this contribution.

However the inference time is incredibly slow, even for a few words it takes about 30 seconds to get a response which makes it unusable.

I am using the small model (I feel the accuracy of the tiny and base model are not up to par in different languages which is the use case in our scenario).

Is there any way to speed up the inference time? Is GPU inference possible on android for example?

For reference, I am using these parameters:

params.print_progress = false;
params.print_special = false;
params.print_realtime = false;
params.print_timestamps = false;
params.translate = false;
params.single_segment = false;
params.max_tokens = 32;
params.language = "auto";
params.n_threads = std::max(1, std::min(8, (int32_t) std::thread::hardware_concurrency()));
params.audio_ctx = 768;
params.speed_up = false;
// params.temperature_inc = params.temperature_inc;
// params.prompt_tokens = nullptr;
params.prompt_n_tokens = 0;

[Azeirah]

GPU inference should be possible with CLBLAST, no? Might try that.

[gersomonline]

Are you running the app in release mode? It improves the performance significantly.

[sandorkonya]

@Azeirah if someone is interested:
Whisper GPU support via CLBLAST
CLBLAST on Android

[sandorkonya]

FYI: maybe this helps.

[ningpengtao-coder]

I try to compile clblast as a shared object of the android platform, use -DGGML_USE_CLBLAST to enable support for clblast, and load opencl through System.load("/vendor/lib64/libOpenCL.so"). Then run inference. Incorrect results and longer inference time.
whisper1(cpu): correct result.

whisper2(opencl gpu): incorrect result.

[ValleZ]

ggerganov/llama.cpp#5123