Add extra arguments to hubert pretrain factory functions #2345
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nateanl
requested review from
mthrok,
hwangjeff,
xiaohui-zhang and
carolineechen
|
@nateanl has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
torchaudio/models/wav2vec2/model.py
Outdated
| @@ -1096,6 +1114,9 @@ def hubert_pretrain_xlarge( | |||
| encoder_ff_interm_dropout: float = 0.0, | |||
| encoder_dropout: float = 0.0, | |||
| encoder_layer_drop: float = 0.0, | |||
| mask_prob: float = 0.8, | |||
| mask_channel_prob: float = 0.0, | |||
| mask_channel_length: float = 10, | |||
| ) -> HuBERTPretrainModel: | |||
| # Overriding the signature so that the return type is correct on Sphinx | |||
| """hubert_pretrain_xlarge(encoder_projection_dropout: float = 0.0, encoder_attention_dropout: float = 0.0, encoder_ff_interm_dropout: float = 0.0, encoder_dropout: float = 0.0, encoder_layer_drop: float = 0.0) -> torchaudio.models.HuBERTPretrainModel | |||
There was a problem hiding this comment.
same as above -- add new params to sphinx signature override
|
|
|
@nateanl has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
carolineechen
approved these changes
Apr 26, 2022
xiaohui-zhang
pushed a commit
to xiaohui-zhang/audio
that referenced
this pull request
May 4, 2022
Summary: In different pre-training and fine-tuning settings, the `mask_prob`, `mask_channel_prob`, and `mask_channel_length` are different. For example, the settings in [pre-training](https://github.com/pytorch/fairseq/blob/main/examples/hubert/config/pretrain/hubert_base_librispeech.yaml#L70) and [fine-tuning](https://github.com/pytorch/fairseq/blob/main/examples/hubert/config/finetune/base_10h.yaml#L69-L73) are different. The motivation is to avoid overfitting when fine-tuning on a small dataset (example: [fine-tune on 10 minutes of audio](https://github.com/pytorch/fairseq/blob/main/examples/wav2vec/config/finetuning/vox_10m.yaml#L57-L59)). This PR adds the required arguments in the factory functions to make them tunable for pre-training and fine-tuning. `mask_length` is set to `10` by default for all cases, hence it's not included in the factory function. Pull Request resolved: pytorch#2345 Reviewed By: carolineechen, xiaohui-zhang Differential Revision: D35845117 Pulled By: nateanl fbshipit-source-id: 0cbb74d09535d189b8258aa8ee0f88779bdb77e7
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In different pre-training and fine-tuning settings, the
mask_prob,mask_channel_prob, andmask_channel_lengthare different. For example, the settings in pre-training and fine-tuning are different. The motivation is to avoid overfitting when fine-tuning on a small dataset (example: fine-tune on 10 minutes of audio).This PR adds the required arguments in the factory functions to make them tunable for pre-training and fine-tuning.
mask_lengthis set to10by default for all cases, hence it's not included in the factory function.