Using whisper english model with language="en" argument

[SimpleVictor]

What's the difference between using the language="en" vs using just the english model and what happens when you use both?

Example

1. Just English model

mlx_whisper.transcribe(
    path_or_hf_repo="models/whisper-medium.en-mlx",
)

2. English model + language argument passed

mlx_whisper.transcribe(
    path_or_hf_repo="models/whisper-medium.en-mlx",
    language="en"
)

3. Generic model

mlx_whisper.transcribe(
    path_or_hf_repo="models/whisper-medium-mlx",
)

4. Generic model + language argument passed

mlx_whisper.transcribe(
    path_or_hf_repo="models/whisper-medium-mlx",
    language="en"
)

[awni]

If the model is not multilingual, then specifying the language is ignored. So case 1 and 2 are the same above.

If the language is unspecified, it will try to automatically detect it. So case 3 will try to automatically detect en but case 4 will force en.

[SimpleVictor]

Thanks for clariying!

Last question: Lets assume our app only cares about en. I'm curious on why there was even a specific en model available instead of just using the base model and attach the language argument. Is there a performance/accuracy boost between the 2 approaches below?

mlx_whisper.transcribe(
    path_or_hf_repo="models/whisper-medium.en-mlx",
)

mlx_whisper.transcribe(
    path_or_hf_repo="models/whisper-medium-mlx",
    language="en"
)

[awni]

It's a good question and I can speculate but I don't really know the answer.. I defer you to the original Whisper repo to ask the authors.

[SimpleVictor]

Oh, good point. Thanks for the suggestion. I was able to find a common discussion thread on this topic here

TLDR

IF you have English audio, then:

The .en models will will outperform the multi-language models.
tiny.en will be better than tiny.
base.en will be better than base.
small.en will be better than small.
medium.en will be better than medium.
A tipping point happened at large though. In that case:

Whisper's multi-lingual model (large) became more accurate than the English-only training.
This is why there's no such thing as "large.en", because it performed worse than large!

[SimpleVictor]

Now that I'm looking back, the thing I just posted didn't necessarily answer the exact question but did help me understand why there wasn't a large.en model in general in case others might find this.

[SimpleVictor]

Created a thread here: openai/whisper#2261