Timestamp issues at the beginning of files when using Whisper Annotations

[scott-vector]

This is a great effort, all at lhotse-speech, so thank you.

When using this to get timestamps from audio with beeps/other noises, the first portion of the file's timestamps do not match up. It begins to sync up around 30-40s in. I've tried to use just the speech, but it still takes the beeps into account. Any suggestions would be great. Thanks.

[pzelasko]

Depending on how noisy your audio is, it might just be difficult to correctly transcribe for the model. Make sure you're using the latest version of Lhotse, as we fixed some timestamp related issues recently. Also if you're using the default settings, you can try medium.en or large version of Whisper for better results.

[pzelasko]

I think the torchaudio workflow already uses the supervision timestamps. It looks like the two best bets are:

• use a model for telephone domain for forced alignment (not sure if a pretrained one is available)
• insert an extra processing step with VAD to cut out the nonspeech region in supervision segments (basically truncate each supervision's start/duration)

BTW the decompose thing is not necessary, you can do:

for cut in cuts_aligned:
  for sup in cut.supervisions:
    print(cut.id, sup.id, sup.alignment["word"])

or if you really want json objects: for d in cuts_aligned.to_dicts(): print(d)

[scott-vector]

@pzelasko thank you for the guidance on reducing the decompose, that helped testing a LOT.

I tried several pipelines, and they all have the timestamps starting at the exact same time in the recording (about 1.2s into it). This is during the ringing, while the speech doesn't actually start until around 15s. In the 5-6 I tested, I received precisely the same offset.

@desh2608, the first segment Whisper identifies covers the entire range (0-22.4s), so I am not sure what it thinks is right or wrong. VAD has issues as well. I've tried a number of different VAD solutions, including silero, and none I've found so far are able to differentiate between the phone's ring and a human speaking (meaning it counts the rings as spoken portions of the call).

[pzelasko]

Probably you need to train/fine-tune your own VAD on this kind of data to make it work...

[scott-vector]

@pzelasko Yes, this is where my mind was headed... custom VAD and probably a telephone-based pipeline for the segmentation.

While we are on the topic of telephone calls, I do see some multi-channel support in this codebase. Can you explain (or is there a document) how lhotse would handle a dual channel phone call?

[pzelasko]

In general you would create a Recording that has an AudioSource pointing to a 2-channel file (e.g. stereo wav/sph) or two AudioSources pointing to separate single channel files. Check the Switchboard recipe for the former, and the AMI recipe for the latter. Then, supervisions also have a field indicating to which channel they correspond (one or more channels for each of them). So you can have supervision with channel=0 to indicate e.g. transcript from the left channel, and channel=1 for transcript of the right channel.

When you create cuts, I am not 100% sure after recent changes from @desh2608, but I think you have the option to create either MultiCuts that can access both channels data or MonoCuts that correspond only to one channel. I think it depends on how you set the channels in the supervisions (e.g. for telephone speech, it makes sense that a supervision is on a single channel only; if you have AB or XY microphone setup, then you would probably have supervision on both channels; etc.).