Code for the paper accepted at Interspeech-Chime' 2018
Our code is implemented in Keras (v2.0.8 with CUDA). To setup, do the following:
If you do not have any Anaconda or Miniconda distribution, head over to their downloads' site before proceeding further.
Clone the repository and create an environment.
git clone https://github.com/DeepLearn-lab/Acoustic-Feature-Fusion_Chime18.git
conda env create -f environment.yaml
This creates an environment named kerasenv2 with all the dependencies installed.
Download the CHiME-Home, Acoustic Scene and LITIS-Rouen datasets and unzip the required datasets.
In the following steps I am going to show how to run the code on the given dataset. (This is what you would need to do for your own audio classification and tagging dataset as well.) The scripts that I am going to mention in each step, please open them and see if you need to the paths.
To extract audio features, run the following:
python feature_extraction.py
The extracted features will get saved at Fe/ and Fe_eva/ folders.
Training with defaut parameters works in the following way
1. Audio Tagging
- python mainfile.py
2. Classification
- python mainfile_classify.py
You can add your own module in models.py
- Audio Tagging
y=[]
y_pred_new = []
class_eer=[]
p_y_pred = lrmodel.predict([X0,X1,X2]) # probability, size: (n_block,label)
for i in range(0,len(p_y_pred),12):
y_pred = np.mean(p_y_pred[i:i+12],axis=0)
y_pred_new.append(y_pred)
y.append(te_y2[i].tolist())
y_pred_new, y = np.array(y_pred_new), np.array(y)
print y_pred_new.shape
print y.shape
n_out = y.shape[1]
print y_pred_new.shape
print y.shape
for k in xrange(n_out):
eer = EER(y[:, k],y_pred_new[:, k])
print "Class ",k,'ERR ',eer
class_eer.append(eer)
EER1=np.mean(class_eer)
print("EER",EER1)
Class 0 ERR 0.14377162629757783
Class 1 ERR 0.20401913875598086
Class 2 ERR 0.2398412698412699
Class 3 ERR 0.12727272727272726
Class 4 ERR 0.1588235294117647
Class 5 ERR 0.06666666666666667
Class 6 ERR 0.03994082840236687
('EER', 0.14019082666405057)
- Classification
y=[]
y_pred_new = []
p_y_pred = lrmodel.predict([X0,X1,X2]) # probability, size: (n_block,label)
for i in range(0,len(p_y_pred),258):
y_pred = np.mean(p_y_pred[i:i+258],axis=0)
y_pred_new.append(y_pred)
y.append(te_y2[i].tolist())
y_pred_new, y = np.array(y_pred_new), np.array(y)
preds = np.argmax( y_pred_new, axis=-1 ) # size: (n_block)
b = scipy.stats.mode(preds) # Finding the maximum value out of the array for predicting the label
pred = int( b[0] )
truth = open(cfg.eval_txt,'r').readlines()
pred.sort()
truth.sort()
pred = [i.split('\t')[1].split('\n')[0]for i in pred]
truth = [i.split('\t')[1]for i in truth]
met = metrics.classification_report(truth,pred)
val = met.split('\n')
print '\n\n'
for i in val:
if len(i)>0:
print i
print '\n\n'
pos,neg=0,0
for i in range(0,len(pred)):
if pred[i] == truth[i]:
pos = pos+1
else:
neg = neg+1
print "\n\n ********************* Accuracy ********************* \n"
print "correctly classified --> ",pos
print "not correctly classified -->",neg
print "percentage(%) accuracy --> ",(float(pos)/float(len(pred)))*100
acc=(float(pos)/float(len(pred)))*100
We have integrated Tensorboard_logger to visualize training and mean squared error. To install tensorboard logger use :
pip install tensorboard_logger
The Training curves for vanilla model and proposed architecture for Tagging Task,
The training loss and Mean Squared Error(MSE) for the attention based system shows a steep decrease in the loss function as compared to the vanilla model.
| Techniques | DCASE | ROUEN (F1) | CHiME-Home |
|---|---|---|---|
| Vanilla | 85.50 | 96.85 | 15.6 |
| EF | 86.1 | 96.36 | 15.0 |
| LF | 87.00 | 96.80 | 14.6 |
| EF+LF | 88.70 | 98.25 | 14.0 |
BSD