Chime4 #423
Chime4 #423
Conversation
change model
remove clipping before ASR
|
I'm undecided about this. But to make it more modular, we should attach the data prep to the dataloader (needs refactoring of the data part of Asteroid, which we want to do), and we should enable evaluating any model on any dataset (also needs refactoring). I'd vote in favour of merging it, because it diversifies the examples that we have (both for the users and things in mind when we refactor). |
|
Yes, I'd say merge it, as it's not introducing any new way of doing things, so it doesn't add much work to our refactoring plate. Did you forget to rename the "chime3" variables and CLI flag or is that on purpose? |
That's on purpose because CHiME 4 uses data from CHiME 3 when you download the data you actually get a directory named CHiME3. But maybe it's confusing. |
|
I guess it will be clear once people have the data from CHiME3 downloaded. |
asteroid/data/chime4_dataset.py
Outdated
| import os | ||
|
|
||
|
|
||
| class CHiME4(Dataset): |
There was a problem hiding this comment.
Let's call this CHiME4Dataset.
asteroid/data/chime4_dataset.py
Outdated
| mixture = torch.from_numpy(mixture) | ||
| mock_source = torch.vstack([mixture]) | ||
| if self.return_id: | ||
| id1 = row.ID | ||
| return mixture, mock_source, [id1] | ||
| return mixture, mock_source |
There was a problem hiding this comment.
Let's not return the mock_source.
egs/chime4/ConvTasNet/eval.py
Outdated
| mix, sources, ids = test_set[idx] | ||
| mix, sources = tensors_to_device([mix, sources], device=model_device) | ||
| est_sources = model(mix.unsqueeze(0)) | ||
| mix_np = mix.cpu().data.numpy() | ||
| sources_np = sources.cpu().data.numpy() | ||
| est_sources_np = est_sources.squeeze(0).cpu().data.numpy() | ||
| est_sources_np *= np.max(np.abs(mix_np)) / np.max(np.abs(est_sources_np)) |
There was a problem hiding this comment.
Remove the code related to sources.
egs/chime4/ConvTasNet/eval.py
Outdated
| utt_metrics.update( | ||
| **wer_tracker( | ||
| mix=mix_np, | ||
| clean=sources_np, |
There was a problem hiding this comment.
Extend WerTracker to work without clean reference. And remove clean.
egs/chime4/ConvTasNet/run.sh
Outdated
| # Choice for the ASR model whether trained on clean or noisy data. One of clean or noisy | ||
| asr_type=noisy | ||
|
|
||
| test_dir=data/test |
There was a problem hiding this comment.
Does this need to be overwritten?
If it doesn't, then add it after utils/parse_options.sh
egs/chime4/ConvTasNet/run.sh
Outdated
| if [[ $stage -le 0 ]]; then | ||
| echo "Stage 0: Generating CHiME-4 dataset" | ||
| $python_path local/create_metadata.py --chime3_dir $storage_dir/CHiME3/ | ||
| fi | ||
|
|
||
| if [[ $stage -le 1 ]]; then | ||
| echo "Stage 2 : Evaluation" | ||
| echo "Results from the following experiment will be stored in $exp_dir/chime4/$asr_type" | ||
|
|
||
| if [[ $compute_wer -eq 1 ]]; then | ||
|
|
||
| # Install espnet if not instaled | ||
| if ! python -c "import espnet" &> /dev/null; then | ||
| echo 'This recipe requires espnet. Installing requirements.' | ||
| $python_path -m pip install espnet_model_zoo | ||
| $python_path -m pip install jiwer | ||
| $python_path -m pip install tabulate | ||
| fi | ||
| fi | ||
|
|
||
| $python_path eval.py \ | ||
| --exp_dir $exp_dir \ | ||
| --test_dir $test_dir \ | ||
| --use_gpu $eval_use_gpu \ | ||
| --compute_wer $compute_wer \ | ||
| --asr_type $asr_type | ||
| fi |
There was a problem hiding this comment.
Fix indents, but otherwise OK
egs/chime4/README.md
Outdated
| As the channel to use for the training set wasn't defined by | ||
| the challenge's rules, we will set it randomly. | ||
|
|
||
| NOTE : |
There was a problem hiding this comment.
| NOTE : | |
| **Note :** |
egs/chime4/README.md
Outdated
|
|
||
| NOTE : | ||
| This dataset uses real noisy data. This means the clean speech from the noisy | ||
| utterances is not available. This makes it not suitable for the usual training |
There was a problem hiding this comment.
| utterances is not available. This makes it not suitable for the usual training | |
| utterances is not available. This makes it unsuitable for the usual training |
| f for f in glob(os.path.join(chime3_dir, "data", "annotations", "*real*.list")) | ||
| ] | ||
| for c3_annot_file_path in c3_annot_files: | ||
| # Read CHiME-3 annotation file |
There was a problem hiding this comment.
I think it could be helpful to move the entire for loop body into a separate function so as to make it obvious that all loop iterations are entirely independent.
| c3_annot_file_path.replace(".json", "_1ch_track.list"), header=None, names=["path"] | ||
| ) | ||
| else: | ||
| c4_annot_file = None |
There was a problem hiding this comment.
This case isn't handled in the create_dataframe function
| # Read CHiME-3 annotation file | ||
| c3_annot_file = pd.read_json(c3_annot_file_path) | ||
| # subsets : "tr" "dt" "et" origin "real" or "simu" | ||
| subset, origin = os.path.split(c3_annot_file_path)[1].replace(".json", "").split("_") |
There was a problem hiding this comment.
I think the entire code could benefit from using the pathlib API. Eg here, c3_annot_file_path.with_suffix("").name.split("_")
Also I think it could be helpful to add a comment with a file name example everywhere filenames are parsed or constructed, for example
# Extract subset and origin from /foo/bar/<subset>_<origin>.json
| ) | ||
| else: | ||
| c4_annot_file = None | ||
| df, df_2 = create_dataframe(chime3_dir, c3_annot_file, c4_annot_file, subset, origin) |
There was a problem hiding this comment.
Could there be better names than df and df_2?
|
|
||
| def create_dataframe(chime3_dir, c3_annot_file, c4_annot_file, subset, origin): | ||
| # Empty list for DataFrame creation | ||
| row_list = [] |
There was a problem hiding this comment.
Again, is there a more helpful name than "row" and "row 2" here?
| env = row.environment | ||
| # if we are not dealing with tr subset | ||
| if "tr" not in subset: | ||
| mixture_path = c4_annot_file[c4_annot_file["path"].str.contains(ID + "_" + env)].values[ |
There was a problem hiding this comment.
Here, and also in the else case below, it could be helpful to add a comment with an example of what we're looking for in the annotation data frame, like the I suggested with the path splitting above
reformat create_metadata.py fix typo indent
fix eval
About this PR
This PR introduces a recipe for evaluation purposes only, on real noisy data.
It uses the data from the CHiME 4 challenge (an adaptation from the CHiME 3 challenge).
Design
I am unsure about the design of the dataset and the recipe :