Questions about get_downsample_rate() in upstream/mockingjay/expert.py

[Sreyan88]

Hi there,

There might be a possible mistake in:

def get_downsample_rate(self):
        return 160

Since each audio is downsampled to 16khz, I think this should be 16?

I have the same doubt with:

example_wavs = [torch.zeros(160000, dtype=torch.float).to(device) for _ in range(16)]

in upstream/mockingjay/example_extract.py. Should it be 16000 instead?

Please let me know if I am making a mistake anywhere. Thank You.

[leo19941227]

16khz is the sample rate of raw audio waveform.
160 is the further downstream rate of mockingjay

[Sreyan88]

Thank you for the clarification! :) Would be great if you provide a link or a small explanation as to how the downstream rate of mockingjay is being used as I am new to this. :)

[andi611]

Hi,

Given a 16kHz audio, every downsample_rate amount of audio samples will be encoded to a representation vector. That is what downsample_rate means here.

I have the same doubt with:
example_wavs = [torch.zeros(160000, dtype=torch.float).to(device) for _ in range(16)]
in upstream/mockingjay/example_extract.py. Should it be 16000 instead?

As such, you can have any positive integer for torch.zeros(any_num, dtype=torch.float).

The downsampling rate is related to the feature preprocessing process itself, and not Mockingjay.
It just happens that we pre-train Mockingjay on features that have a downsampling rate of 160, as is usually done for reconstruction methods (for example APC, VQ-APC, NPC, etc).
Mockingjay will generate one representation per input acoustic frame.

I will close this for now since there is no mistake.
You can ask more follow-up questions in this thread if needed.

Andy

[Sreyan88]

Thank you so much for the great explanation @andi611 ! I am constantly reading through the repo/paper with a motive of coming up with a pull request soon! Will post more questions soon on the same thread! :)

[andi611]

Thank you so much for the great explanation @andi611 ! I am constantly reading through the repo/paper with a motive of coming up with a pull request soon! Will post more questions soon on the same thread! :)

That sounds really nice! We look forward to your pull request!