chatbot_korean.md

Results

Train/Test with Complete • Test with Incomplete • Train/Test with Incomplete

Complete

• Epoch 100 (bs8)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs8	91.51	90.57	94.34	94.34	96.23	89.62	91.51	93.40	93.40	98.11	93.30	2.48
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs8	93.40	93.40	95.28	100.00	93.40	95.28	92.45	94.34	96.23	95.28	94.91	2.03
BERTwithAttClsProjection-bs8	99.06	96.23	96.23	91.51	97.17	98.11	95.28	97.17	94.34	96.23	96.13	2.00
BERTwithProjection-bs8	95.28	91.51	97.17	90.57	98.11	95.28	93.40	99.06	92.45	94.34	94.72	2.67
BERTwithProjectionAtt-bs8	95.28	91.51	93.40	95.28	97.17	90.57	92.45	94.34	93.40	92.45	93.58	1.88
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8	93.40	99.06	92.45	94.34	91.51	89.62	95.28	93.40	97.17	99.06	94.53	2.98
EmbraceBERT-bs8-p_attclsqw	94.34	91.51	93.40	95.28	93.40	90.57	92.45	94.34	91.51	92.45	92.92	1.42
EmbraceBERTwithProj-bs8	97.17	97.17	98.11	96.23	98.11	98.11	97.17	97.17	98.11	97.17	97.45	0.60
EmbraceBERTwithProj-bs8-p_attclsqw	97.17	99.06	95.28	95.28	99.06	98.11	97.17	97.17	96.23	97.17	97.17	1.27
EmbraceBERTconcatAtt-bs8	96.23	94.34	96.23	93.40	89.62	93.40	90.57	97.17	98.11	97.17	94.62	2.74
EmbraceBERTconcatAtt-bs8-p_attclsqw	95.28	94.34	95.28	96.23	92.45	88.68	98.11	95.28	94.34	96.23	94.62	2.43
EmbraceBERTconcatAttwithProj-bs8	98.11	96.23	89.62	95.28	97.17	96.23	98.11	94.34	93.40	96.23	95.47	2.42
EmbraceBERTconcatAttwithProj-bs8-p_attclsqw	97.17	97.17	98.11	97.17	96.23	96.23	98.11	99.06	94.34	96.23	96.98	1.25
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-KVQ-bs8	97.17	88.68	96.23	91.51	91.51	94.34	92.45	89.62	92.45	93.40	92.74	2.53
EmbraceBERT-KVQ-bs8-p_attclsqw	88.68	95.28	91.51	88.68	95.28	96.23	92.45	95.28	94.34	91.51	92.92	2.64
EmbraceBERT-KVQ-withProj-bs8	98.11	98.11	98.11	99.06	99.06	97.17	97.17	100.00	98.11	99.06	98.40	0.85
EmbraceBERT-KVQ-withProj-bs8-p_attclsqw	97.17	97.17	96.23	98.11	95.28	99.06	98.11	96.23	93.40	99.06	99.06	97.17
EmbraceBERTconcatAtt-KVQ-bs8	93.40	95.28	94.34	95.28	95.28	88.68	94.34	94.34	94.34	94.34	93.96	1.85
EmbraceBERTconcatAtt-KVQ-bs8-p_attclsqw	97.17	96.23	97.17	96.23	93.40	97.17	95.28	92.45	90.57	94.34	95.00	2.15
EmbraceBERTconcatAtt-KVQ-withProj-bs8	95.28	95.28	97.17	91.51	94.34	96.23	96.23	96.23	98.11	95.28	95.57	1.69
EmbraceBERTconcatAtt-KVQ-withProj-bs8-p_attclsqw	98.11	95.28	94.34	97.17	98.11	97.17	94.34	91.51	91.51	97.17	95.47	2.38

STT Error: trained with COMPLETE data and tested with INCOMPLETE data

gtts-google

• Epoch 100 (bs8)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs8	89.62	87.74	85.85	70.75	89.62	88.68	69.81	89.62	87.74	87.74	84.72	7.31
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs8	88.68	88.68	87.74	91.51	68.87	88.68	88.68	87.74	89.62	68.87	84.91	8.08
BERTwithAttClsProjection-bs8	71.70	88.68	88.68	87.74	69.81	88.68	89.62	89.62	70.75	87.74	83.30	8.25
BERTwithProjection-bs8	88.68	50.00	88.68	83.02	84.91	70.75	89.62	69.81	58.49	87.74	77.17	13.45
BERTwithProjectionAtt-bs8	86.79	87.74	88.68	88.68	70.75	70.75	68.87	84.91	90.57	88.68	82.64	8.32
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8	88.68	71.70	86.79	84.91	71.70	86.79	85.85	84.91	64.15	89.62	81.51	8.42
EmbraceBERT-bs8-p_attclsqw	88.68	90.57	66.04	83.02	70.75	69.81	91.51	87.74	65.09	87.74	80.09	10.27
EmbraceBERTwithProj-bs8	88.68	87.74	68.87	87.74	87.74	88.68	87.74	88.68	87.74	87.74	86.13	5.77
EmbraceBERTwithProj-bs8-p_attclsqw	71.70	68.87	88.68	69.81	88.68	69.81	87.74	88.68	87.74	88.68	81.04	9.00
EBERTconcatAtt-bs8	89.62	86.79	88.68	87.74	68.87	86.79	87.74	87.74	85.85	70.75	84.06	7.20
EBERTconcatAtt-bs8_attclsqw	86.79	86.79	89.62	71.70	88.68	69.81	88.68	69.81	88.68	70.75	81.13	8.72
EBERTconcatAttwithProj-bs8	89.62	89.62	68.87	69.81	86.79	87.74	71.70	88.68	70.75	88.68	81.23	8.99
EBERTconcatAttwithProj-bs8_attclsqw	70.75	87.74	88.68	70.75	87.74	87.74	88.68	71.70	90.57	88.68	83.30	8.05
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery	87.74	87.74	89.62	87.74	68.87	88.68	86.79	84.91	86.79	88.68	85.75	5.76
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	86.79	69.81	71.70	83.96	88.68	83.96	88.68	84.91	87.74	80.19	82.64	6.44
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery	87.74	88.68	88.68	88.68	69.81	88.68	67.92	67.92	88.68	88.68	82.55	9.18
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	88.68	88.68	88.68	72.64	88.68	87.74	64.15	88.68	68.87	88.68	82.55	9.36
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery	81.13	88.68	88.68	85.85	68.87	84.91	68.87	68.87	85.85	88.68	81.04	8.25
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	86.79	76.42	88.68	68.87	86.79	67.92	87.74	68.87	88.68	89.62	81.04	8.89
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery	88.68	89.62	86.79	88.68	88.68	88.68	88.68	88.68	89.62	87.74	88.58	0.78
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	87.74	67.92	87.74	88.68	86.79	83.02	71.70	87.74	88.68	87.74	83.77	7.19

gtts-sphinx

• Epoch 100 (bs8)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs8	91.51	94.34	75.47	92.45	96.23	95.28	90.57	97.17	93.40	93.40	91.98	5.83
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs8	91.51	96.23	94.34	87.74	88.68	93.40	94.34	94.34	96.23	88.68	92.55	3.03
BERTwithAttClsProjection-bs8	96.23	91.51	90.57	95.28	86.79	95.28	96.23	97.17	93.40	92.45	93.49	3.06
BERTwithProjection-bs8	95.28	76.42	93.40	82.08	92.45	90.57	94.34	89.62	86.79	96.23	89.72	6.00
BERTwithProjectionAtt-bs8	83.96	92.45	95.28	95.28	95.28	83.96	92.45	85.85	93.40	85.85	90.38	4.62
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8	94.34	87.74	86.79	77.36	95.28	95.28	91.51	88.68	79.25	89.62	88.58	5.91
EmbraceBERT-bs8-p_attclsqw	95.28	97.17	83.96	76.42	92.45	87.74	89.62	89.62	86.79	93.40	89.25	5.73
EmbraceBERTwithProj-bs8	97.17	94.34	93.40	93.40	89.62	90.57	94.34	97.17	96.23	95.28	94.15	2.42
EmbraceBERTwithProj-bs8-p_attclsqw	93.40	81.13	96.23	94.34	93.40	86.79	95.28	96.23	95.28	95.28	92.74	4.66
EBERTconcatAtt-bs8	94.34	85.85	90.57	92.45	94.34	85.85	89.62	93.40	83.96	95.28	90.57	3.89
EBERTconcatAtt-bs8_attclsqw	93.40	89.62	96.23	88.68	92.45	87.74	86.79	72.64	92.45	94.34	89.43	6.30
EBERTconcatAttwithProj-bs8	94.34	97.17	73.58	91.51	92.45	95.28	94.34	95.28	93.40	93.40	92.08	6.35
EBERTconcatAttwithProj-bs8_attclsqw	94.34	95.28	91.51	86.79	96.23	87.74	93.40	96.23	92.45	95.28	92.92	3.19
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery	92.45	85.85	92.45	91.51	82.08	92.45	93.40	85.85	83.02	91.51	89.06	4.14
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	82.08	89.62	90.57	88.68	90.57	93.40	91.51	93.40	96.23	86.79	90.28	3.73
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery	94.34	92.45	95.28	90.57	93.40	94.34	95.28	83.02	95.28	94.34	92.83	3.56
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	94.34	94.34	94.34	92.45	96.23	97.17	80.19	94.34	88.68	95.28	92.74	4.72
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery	80.19	93.40	93.40	86.79	93.40	92.45	83.02	77.36	90.57	86.79	87.74	5.61
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	93.40	69.81	96.23	66.04	93.40	89.62	93.40	94.34	86.79	94.34	87.74	10.26
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery	92.45	96.23	92.45	94.34	95.28	93.40	94.34	95.28	96.23	96.23	94.62	1.40
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	94.34	95.28	91.51	94.34	96.23	84.91	97.17	97.17	95.28	98.11	94.43	3.64

gtts-witai

• Epoch 100 (bs8)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs8	92.45	88.68	84.91	93.40	94.34	89.62	90.57	95.28	88.68	96.23	91.42	3.36
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs8	89.62	92.45	94.34	91.51	89.62	93.40	93.40	91.51	94.34	88.68	91.89	1.94
BERTwithAttClsProjection-bs8	97.17	94.34	94.34	88.68	88.68	95.28	95.28	97.17	92.45	94.34	93.77	2.87
BERTwithProjection-bs8	92.45	82.08	94.34	87.74	90.57	91.51	93.40	96.23	83.96	88.68	90.09	4.29
BERTwithProjectionAtt-bs8	88.68	87.74	90.57	94.34	95.28	89.62	89.62	87.74	89.62	88.68	90.19	2.47
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8	90.57	92.45	87.74	86.79	92.45	90.57	92.45	86.79	84.91	95.28	90.00	3.13
EmbraceBERT-bs8-p_attclsqw	94.34	94.34	85.85	84.91	92.45	83.02	94.34	90.57	84.91	91.51	89.62	4.26
EmbraceBERTwithProj-bs8	96.23	92.45	95.28	94.34	95.28	95.28	95.28	96.23	95.28	95.28	95.09	1.02
EmbraceBERTwithProj-bs8-p_attclsqw	95.28	90.57	95.28	95.28	94.34	94.34	94.34	96.23	94.34	96.23	94.62	1.52
EBERTconcatAtt-bs8	95.28	87.74	95.28	88.68	87.74	91.51	88.68	94.34	92.45	95.28	91.70	3.09
EBERTconcatAtt-bs8_attclsqw	93.40	94.34	92.45	93.40	93.40	83.02	92.45	80.19	95.28	92.45	91.04	4.83
EBERTconcatAttwithProj-bs8	94.34	97.17	78.30	89.62	91.51	96.23	94.34	92.45	88.68	97.17	91.98	5.36
EBERTconcatAttwithProj-bs8_attclsqw	96.23	95.28	96.23	92.45	95.28	89.62	95.28	96.23	92.45	93.40	94.25	2.09
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery	93.40	85.85	95.28	88.68	86.79	95.28	92.45	87.74	94.34	93.40	91.32	3.47
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	90.57	87.74	86.79	89.62	91.51	86.79	94.34	92.45	94.34	79.25	89.34	4.28
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery	93.40	95.28	95.28	96.23	98.11	94.34	95.28	91.51	96.23	94.34	95.00	1.69
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	94.34	95.28	96.23	91.51	97.17	95.28	85.85	93.40	93.40	96.23	93.87	3.11
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery	83.02	90.57	90.57	88.68	90.57	83.96	87.74	84.91	92.45	90.57	88.30	3.11
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	93.40	82.08	92.45	86.79	91.51	88.68	92.45	93.40	93.40	95.28	90.94	3.78
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery	92.45	94.34	90.57	91.51	91.51	95.28	93.40	96.23	97.17	95.28	93.77	2.12
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	93.40	93.40	93.40	96.23	97.17	86.79	96.23	94.34	94.34	96.23	94.15	2.79

macsay-google

• Epoch 100 (bs8)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs8	83.02	82.08	59.43	86.79	89.62	93.40	76.42	92.45	83.02	88.68	83.49	9.41
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs8	84.91	91.51	92.45	83.02	68.87	92.45	93.40	86.79	91.51	74.53	85.94	7.97
BERTwithAttClsProjection-bs8	95.28	89.62	90.57	88.68	69.81	94.34	94.34	92.45	84.91	87.74	88.77	7.05
BERTwithProjection-bs8	90.57	70.75	93.40	68.87	82.08	87.74	92.45	88.68	68.87	83.02	82.64	9.26
BERTwithProjectionAtt-bs8	73.58	77.36	88.68	92.45	90.57	77.36	86.79	68.87	84.91	77.36	81.79	7.52
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8	89.62	74.53	68.87	66.04	72.64	91.51	76.42	84.91	57.55	89.62	77.17	10.87
EmbraceBERT-bs8-p_attclsqw	81.13	81.13	58.49	69.81	78.30	66.04	86.79	90.57	67.92	82.08	76.23	9.66
EmbraceBERTwithProj-bs8	93.40	80.19	83.96	93.40	83.02	92.45	93.40	93.40	93.40	92.45	89.91	5.01
EmbraceBERTwithProj-bs8-p_attclsqw	87.74	67.92	86.79	86.79	83.96	83.96	89.62	94.34	93.40	90.57	86.51	7.04
EBERTconcatAtt-bs8	80.19	83.02	88.68	87.74	86.79	80.19	83.96	90.57	74.53	90.57	84.62	4.96
EBERTconcatAtt-bs8_attclsqw	83.96	86.79	92.45	82.08	85.85	64.15	81.13	49.06	91.51	80.19	79.72	12.64
EBERTconcatAttwithProj-bs8	94.34	92.45	52.83	77.36	75.47	90.57	88.68	91.51	77.36	92.45	83.30	12.24
EBERTconcatAttwithProj-bs8_attclsqw	92.45	93.40	90.57	70.75	93.40	73.58	89.62	94.34	87.74	89.62	87.55	7.96
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery	90.57	76.42	80.19	84.91	60.38	90.57	83.02	66.98	86.79	92.45	81.23	10.05
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	70.75	74.53	79.25	76.42	77.36	72.64	95.28	79.25	93.40	69.81	78.87	8.33
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery	89.62	85.85	92.45	93.40	89.62	86.79	87.74	71.70	89.62	93.40	88.02	5.98
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	90.57	94.34	88.68	73.58	94.34	88.68	65.09	89.62	81.13	93.40	85.94	9.26
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery	63.21	79.25	89.62	69.81	78.30	74.53	68.87	61.32	86.79	89.62	76.13	9.87
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	76.42	53.77	77.36	65.09	75.47	72.64	90.57	80.19	77.36	89.62	75.85	10.21
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery	92.45	90.57	73.58	86.79	88.68	91.51	92.45	89.62	92.45	93.40	89.15	5.54
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	92.45	86.79	88.68	92.45	92.45	60.38	93.40	93.40	90.57	92.45	88.30	9.53

macsay-sphinx

• Epoch 100 (bs8)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs8	66.04	66.98	55.66	69.81	74.53	67.92	72.64	73.58	73.58	74.53	69.53	5.53
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs8	73.58	73.58	70.75	62.26	66.98	70.75	70.75	69.81	69.81	66.04	69.43	3.30
BERTwithAttClsProjection-bs8	76.42	69.81	71.70	70.75	70.75	72.64	77.36	69.81	66.04	71.70	71.70	3.10
BERTwithProjection-bs8	69.81	66.04	75.47	64.15	67.92	72.64	76.42	73.58	67.92	71.70	70.57	3.86
BERTwithProjectionAtt-bs8	55.66	66.98	68.87	76.42	70.75	58.49	68.87	66.98	69.81	61.32	66.42	5.89
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8	70.75	57.55	58.49	53.77	68.87	75.47	65.09	66.04	54.72	73.58	64.43	7.48
EmbraceBERT-bs8-p_attclsqw	65.09	67.92	58.49	53.77	68.87	68.87	72.64	69.81	66.98	68.87	66.13	5.43
EmbraceBERTwithProj-bs8	76.42	69.81	72.64	76.42	70.75	76.42	74.53	74.53	75.47	75.47	74.25	2.27
EmbraceBERTwithProj-bs8-p_attclsqw	72.64	59.43	77.36	71.70	71.70	70.75	75.47	75.47	74.53	72.64	72.17	4.68
EBERTconcatAtt-bs8	62.26	59.43	74.53	70.75	69.81	66.98	68.87	72.64	62.26	74.53	68.21	5.08
EBERTconcatAtt-bs8_attclsqw	71.70	68.87	72.64	64.15	73.58	59.43	63.21	51.89	75.47	69.81	67.08	6.99
EBERTconcatAttwithProj-bs8	73.58	75.47	40.57	62.26	67.92	70.75	71.70	72.64	68.87	70.75	67.45	9.60
EBERTconcatAttwithProj-bs8_attclsqw	73.58	73.58	76.42	64.15	71.70	67.92	72.64	74.53	69.81	69.81	71.42	3.40
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery	75.47	61.32	68.87	74.53	54.72	69.81	76.42	71.70	64.15	70.75	68.77	6.51
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	57.55	68.87	71.70	73.58	63.21	68.87	73.58	73.58	69.81	69.81	69.06	4.86
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery	69.81	70.75	75.47	71.70	74.53	70.75	72.64	60.38	73.58	73.58	71.32	4.03
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	76.42	70.75	72.64	75.47	73.58	76.42	64.15	66.98	72.64	72.64	72.17	3.78
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery	54.72	73.58	74.53	63.21	69.81	65.09	57.55	48.11	68.87	69.81	64.53	8.20
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	68.87	44.34	69.81	48.11	57.55	66.98	71.70	71.70	66.04	70.75	63.58	9.56
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery	72.64	70.75	62.26	72.64	71.70	66.98	70.75	69.81	72.64	71.70	70.19	3.11
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	73.58	72.64	67.92	74.53	73.58	58.49	74.53	69.81	75.47	73.58	71.42	4.83

macsay-witai

• Epoch 100 (bs8)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs8	88.68	87.74	79.25	94.34	91.51	90.57	89.62	94.34	83.96	91.51	89.15	4.41
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs8	86.79	89.62	94.34	88.68	82.08	92.45	92.45	85.85	92.45	84.91	88.96	3.80
BERTwithAttClsProjection-bs8	95.28	91.51	94.34	88.68	82.08	96.23	94.34	96.23	92.45	95.28	92.64	4.17
BERTwithProjection-bs8	91.51	82.08	94.34	86.79	87.74	90.57	94.34	95.28	79.25	86.79	88.87	5.09
BERTwithProjectionAtt-bs8	83.96	83.02	85.85	94.34	90.57	91.51	92.45	81.13	87.74	88.68	87.92	4.15
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8	92.45	85.85	80.19	82.08	85.85	89.62	86.79	86.79	72.64	93.40	85.57	5.80
EmbraceBERT-bs8-p_attclsqw	86.79	88.68	80.19	79.25	90.57	77.36	93.40	91.51	79.25	86.79	85.38	5.57
EmbraceBERTwithProj-bs8	95.28	87.74	86.79	96.23	90.57	95.28	94.34	96.23	94.34	93.40	93.02	3.27
EmbraceBERTwithProj-bs8-p_attclsqw	91.51	84.91	94.34	91.51	94.34	91.51	89.62	94.34	94.34	91.51	91.79	2.80
EBERTconcatAtt-bs8	92.45	84.91	94.34	93.40	91.51	90.57	86.79	93.40	88.68	94.34	91.04	3.11
EBERTconcatAtt-bs8_attclsqw	87.74	92.45	93.40	90.57	87.74	79.25	91.51	72.64	97.17	83.02	87.55	6.98
EBERTconcatAttwithProj-bs8	94.34	94.34	68.87	83.02	86.79	93.40	92.45	91.51	84.91	94.34	88.40	7.63
EBERTconcatAttwithProj-bs8_attclsqw	95.28	94.34	95.28	83.02	96.23	83.02	90.57	96.23	85.85	90.57	91.04	5.07
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery	94.34	86.79	89.62	85.85	74.53	94.34	88.68	83.02	91.51	93.40	88.21	5.80
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	85.85	80.19	81.13	84.91	89.62	82.08	93.40	85.85	95.28	77.36	85.57	5.49
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery	92.45	86.79	93.40	92.45	91.51	88.68	92.45	90.57	92.45	94.34	91.51	2.15
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	94.34	93.40	94.34	85.85	96.23	91.51	80.19	90.57	87.74	96.23	91.04	4.87
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery	78.30	88.68	87.74	81.13	85.85	81.13	80.19	77.36	87.74	92.45	84.06	4.82
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	89.62	73.58	86.79	82.08	83.96	84.91	93.40	84.91	88.68	93.40	86.13	5.53
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery	91.51	93.40	85.85	86.79	90.57	93.40	92.45	94.34	94.34	90.57	91.32	2.82
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	92.45	90.57	90.57	95.28	93.40	78.30	96.23	93.40	93.40	95.28	91.89	4.87

STT Error: trained and tested with INCOMPLETE data

gtts-google

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs8	83.96	88.68	85.85	85.85	86.79	87.74	89.62	90.57	86.79	85.85	87.17	1.90
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs8	87.74	84.91	88.68	88.68	88.68	87.74	87.74	86.79	87.74	82.08	87.08	1.98
BERTwithAttClsProjection-bs8	87.74	84.91	87.74	86.79	83.96	89.62	85.85	85.85	85.85	84.91	86.32	1.59
BERTwithProjection-bs8	87.74	80.19	87.74	87.74	86.79	86.79	86.79	88.68	85.85	87.74	86.60	2.26
BERTwithProjectionAtt-bs8	87.74	85.85	88.68	85.85	85.85	86.79	89.62	85.85	88.68	85.85	87.08	1.40
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8	88.68	83.96	90.57	88.68	90.57	85.85	84.91	87.74	84.91	88.68	87.45	2.27
EmbraceBERT-bs8-p_attclsqw	89.62	88.68	86.79	86.79	86.79	86.79	87.74	86.79	86.79	86.79	87.36	0.96
EmbraceBERTwithProj-bs8	87.74	87.74	87.74	88.68	87.74	86.79	87.74	87.74	88.68	88.68	87.92	0.57
EmbraceBERTwithProj-bs8-p_attclsqw	87.74	88.68	89.62	89.62	86.79	88.68	87.74	87.74	90.57	88.68	88.58	1.07
EBERTconcatAtt-bs8	86.79	88.68	86.79	85.85	86.79	85.85	87.74	87.74	88.68	86.79	87.17	0.96
EBERTconcatAtt-bs8_attclsqw	86.79	87.74	86.79	88.68	86.79	85.85	89.62	86.79	84.91	87.74	87.17	1.28
EBERTconcatAttwithProj-bs8	87.74	87.74	86.79	85.85	83.96	88.68	88.68	88.68	87.74	87.74	87.36	1.41
EBERTconcatAttwithProj-bs8_attclsqw	86.79	87.74	83.96	88.68	88.68	87.74	88.68	88.68	88.68	86.79	87.64	1.43
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery	85.85	87.74	88.68	85.85	87.74	83.96	87.74	87.74	87.74	85.85	86.89	1.36
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	87.74	87.74	88.68	84.91	85.85	85.85	85.85	86.79	86.79	86.79	86.70	1.07
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery	87.74	88.68	88.68	86.79	89.62	88.68	86.79	88.68	89.62	87.74	88.30	0.96
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	87.74	88.68	86.79	87.74	87.74	89.62	87.74	88.68	87.74	89.62	88.21	0.87
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery	88.68	83.96	86.79	88.68	87.74	84.91	85.85	86.79	87.74	88.68	86.98	1.57
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	87.74	89.62	87.74	88.68	85.85	89.62	85.85	88.68	87.74	86.79	87.83	1.30
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery	86.79	86.79	87.74	85.85	88.68	85.85	87.74	89.62	87.74	83.96	87.08	1.52
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	85.85	86.79	84.91	88.68	85.85	86.79	88.68	87.74	84.91	88.68	86.89	1.43

gtts-sphinx

• Epoch 100 (bs8)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs8	91.51	92.45	95.28	95.28	95.28	95.28	95.28	92.45	94.34	94.34	94.15	1.39
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs8	92.45	95.28	95.28	92.45	92.45	94.34	95.28	93.40	92.45	95.28	93.87	1.28
BERTwithAttClsProjection-bs8	96.23	95.28	95.28	90.57	94.34	91.51	96.23	95.28	95.28	93.40	94.34	1.84
BERTwithProjection-bs8	96.23	82.08	97.17	87.74	93.40	91.51	95.28	93.40	97.17	93.40	92.74	4.47
BERTwithProjectionAtt-bs8	90.57	94.34	94.34	89.62	96.23	88.68	93.40	94.34	93.40	93.40	92.83	2.28
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8	93.40	92.45	95.28	93.40	94.34	92.45	94.34	92.45	91.51	92.45	93.21	1.10
EmbraceBERT-bs8-p_attclsqw	89.62	95.28	96.23	86.79	95.28	94.34	93.40	87.74	96.23	95.28	93.02	3.41
EmbraceBERTwithProj-bs8	93.40	94.34	94.34	93.40	90.57	91.51	95.28	95.28	94.34	90.57	93.30	1.71
EmbraceBERTwithProj-bs8-p_attclsqw	93.40	94.34	90.57	93.40	90.57	94.34	93.40	96.23	97.17	96.23	93.96	2.12
EBERTconcatAtt-bs8	93.40	96.23	94.34	95.28	91.51	94.34	93.40	92.45	85.85	95.28	93.21	2.79
EBERTconcatAtt-bs8_attclsqw	93.40	94.34	93.40	94.34	94.34	88.68	87.74	91.51	92.45	95.28	92.55	2.40
EBERTconcatAttwithProj-bs8	92.45	96.23	96.23	95.28	92.45	92.45	94.34	92.45	96.23	94.34	94.25	1.60
EBERTconcatAttwithProj-bs8_attclsqw	94.34	95.28	96.23	95.28	93.40	90.57	94.34	95.28	93.40	94.34	94.25	1.49
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery	93.40	92.45	92.45	89.62	91.51	94.34	89.62	92.45	92.45	84.91	91.32	2.56
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	96.23	93.40	93.40	94.34	88.68	95.28	85.85	93.40	90.57	94.34	92.55	3.06
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery	93.40	95.28	96.23	92.45	91.51	88.68	93.40	95.28	94.34	93.40	93.40	2.07
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	92.45	94.34	93.40	93.40	97.17	96.23	94.34	91.51	93.40	93.40	93.96	1.59
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery	94.34	94.34	93.40	91.51	94.34	95.28	95.28	93.40	95.28	95.28	94.25	1.15
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	96.23	94.34	96.23	91.51	94.34	96.23	96.23	95.28	86.79	95.28	94.25	2.84
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery	93.40	91.51	96.23	93.40	90.57	95.28	94.34	92.45	93.40	94.34	93.49	1.60
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	95.28	93.40	95.28	96.23	94.34	93.40	94.34	93.40	94.34	95.28	94.53	0.92

gtts-witai

• Epoch 100 (bs8)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs8	94.34	92.45	94.34	92.45	92.45	93.40	92.45	94.34	92.45	92.45	93.11	0.85
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs8	93.40	93.40	92.45	93.40	94.34	93.40	92.45	95.28	95.28	92.45	93.58	1.02
BERTwithAttClsProjection-bs8	95.28	93.40	93.40	95.28	94.34	95.28	93.40	94.34	94.34	93.40	94.25	0.78
BERTwithProjection-bs8	92.45	89.62	91.51	93.40	95.28	92.45	94.34	91.51	91.51	93.40	92.55	1.55
BERTwithProjectionAtt-bs8	94.34	93.40	94.34	93.40	92.45	93.40	94.34	91.51	92.45	92.45	93.21	0.92
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8	92.45	94.34	92.45	92.45	93.40	92.45	92.45	90.57	91.51	92.45	92.45	0.94
EmbraceBERT-bs8-p_attclsqw	93.40	94.34	92.45	91.51	92.45	92.45	92.45	94.34	92.45	92.45	92.83	0.86
EmbraceBERTwithProj-bs8	94.34	94.34	94.34	93.40	96.23	94.34	95.28	94.34	94.34	94.34	94.53	0.71
EmbraceBERTwithProj-bs8-p_attclsqw	94.34	94.34	94.34	95.28	91.51	94.34	95.28	94.34	93.40	94.34	94.15	1.02
EBERTconcatAtt-bs8	93.40	92.45	92.45	93.40	93.40	93.40	92.45	94.34	91.51	93.40	93.02	0.75
EBERTconcatAtt-bs8_attclsqw	91.51	93.40	90.57	94.34	93.40	94.34	95.28	91.51	92.45	92.45	92.92	1.42
EBERTconcatAttwithProj-bs8	94.34	94.34	94.34	92.45	95.28	94.34	96.23	94.34	91.51	91.51	93.87	1.48
EBERTconcatAttwithProj-bs8_attclsqw	96.23	94.34	94.34	94.34	94.34	94.34	94.34	95.28	94.34	93.40	94.53	0.71
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery	91.51	92.45	94.34	93.40	93.40	92.45	94.34	93.40	94.34	92.45	93.21	0.92
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	93.40	90.57	93.40	93.40	94.34	93.40	93.40	93.40	93.40	91.51	93.02	1.05
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery	94.34	94.34	94.34	94.34	94.34	94.34	94.34	95.28	94.34	94.34	94.43	0.28
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	94.34	94.34	94.34	92.45	95.28	93.40	94.34	93.40	94.34	94.34	94.06	0.74
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery	93.40	94.34	93.40	92.45	93.40	95.28	93.40	93.40	92.45	90.57	93.21	1.18
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	92.45	92.45	93.40	91.51	91.51	93.40	94.34	91.51	91.51	93.40	92.55	0.98
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery	92.45	94.34	96.23	92.45	94.34	95.28	93.40	93.40	94.34	92.45	93.87	1.21
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	94.34	93.40	94.34	94.34	94.34	95.28	94.34	93.40	94.34	94.34	94.25	0.51

macsay-google

• Epoch 100 (bs8)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs8	95.28	95.28	97.17	95.28	97.17	95.28	96.23	97.17	95.28	96.23	96.04	0.82
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs8	96.23	97.17	96.23	96.23	97.17	95.28	96.23	97.17	95.28	97.17	96.42	0.71
BERTwithAttClsProjection-bs8	96.23	97.17	94.34	95.28	96.23	96.23	93.40	95.28	97.17	95.28	95.66	1.13
BERTwithProjection-bs8	95.28	96.23	96.23	93.40	93.40	92.45	97.17	95.28	95.28	95.28	95.00	1.40
BERTwithProjectionAtt-bs8	95.28	96.23	97.17	96.23	93.40	94.34	91.51	96.23	94.34	94.34	94.91	1.59
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8	95.28	96.23	94.34	94.34	96.23	94.34	91.51	95.28	95.28	95.28	94.81	1.28
EmbraceBERT-bs8-p_attclsqw	95.28	97.17	96.23	94.34	97.17	95.28	95.28	95.28	95.28	96.23	95.75	0.87
EmbraceBERTwithProj-bs8	93.40	95.28	96.23	94.34	95.28	96.23	95.28	93.40	95.28	95.28	95.00	0.95
EmbraceBERTwithProj-bs8-p_attclsqw	95.28	93.40	92.45	93.40	95.28	95.28	95.28	93.40	96.23	95.28	94.53	1.18
EBERTconcatAtt-bs8	96.23	93.40	96.23	95.28	95.28	96.23	96.23	94.34	95.28	96.23	95.47	0.92
EBERTconcatAtt-bs8_attclsqw	95.28	95.28	97.17	95.28	95.28	97.17	94.34	92.45	96.23	95.28	95.38	1.30
EBERTconcatAttwithProj-bs8	95.28	95.28	97.17	97.17	91.51	97.17	95.28	96.23	95.28	96.23	95.66	1.59
EBERTconcatAttwithProj-bs8_attclsqw	96.23	94.34	93.40	93.40	95.28	97.17	94.34	97.17	96.23	97.17	95.47	1.45
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery	96.23	95.28	95.28	97.17	95.28	93.40	96.23	97.17	94.34	97.17	95.75	1.21
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	94.34	95.28	95.28	95.28	92.45	94.34	94.34	96.23	95.28	94.34	94.72	0.96
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery	94.34	95.28	95.28	94.34	96.23	93.40	95.28	92.45	95.28	95.28	94.72	1.05
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	95.28	94.34	93.40	95.28	95.28	92.45	95.28	95.28	97.17	93.40	94.72	1.28
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery	96.23	94.34	94.34	96.23	95.28	95.28	95.28	94.34	95.28	97.17	95.38	0.89
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	96.23	97.17	97.17	94.34	97.17	96.23	97.17	97.17	96.23	97.17	96.60	0.86
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery	97.17	97.17	96.23	95.28	96.23	94.34	95.28	94.34	93.40	96.23	95.57	1.20
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	96.23	96.23	97.17	97.17	97.17	95.28	96.23	95.28	97.17	95.28	96.32	0.78

macsay-sphinx

• Epoch 100 (bs8)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs8	80.19	76.42	79.25	77.36	78.30	79.25	75.47	76.42	75.47	79.25	77.74	1.64
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs8	74.53	76.42	77.36	76.42	73.58	76.42	74.53	75.47	77.36	82.08	76.42	2.23
BERTwithAttClsProjection-bs8	77.36	77.36	82.08	83.96	80.19	72.64	78.30	73.58	76.42	77.36	77.92	3.30
BERTwithProjection-bs8	81.13	75.47	74.53	73.58	80.19	73.58	79.25	76.42	77.36	77.36	76.89	2.54
BERTwithProjectionAtt-bs8	78.30	72.64	79.25	77.36	77.36	81.13	75.47	80.19	76.42	78.30	77.64	2.31
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8	80.19	75.47	77.36	76.42	79.25	74.53	81.13	75.47	73.58	75.47	76.89	2.40
EmbraceBERT-bs8-p_attclsqw	78.30	80.19	74.53	77.36	84.91	80.19	75.47	77.36	74.53	75.47	77.83	3.08
EmbraceBERTwithProj-bs8	74.53	78.30	73.58	77.36	75.47	75.47	77.36	82.08	80.19	77.36	77.17	2.45
EmbraceBERTwithProj-bs8-p_attclsqw	78.30	83.96	77.36	75.47	83.96	70.75	78.30	77.36	78.30	79.25	78.30	3.63
EBERTconcatAtt-bs8	74.53	70.75	73.58	77.36	73.58	76.42	80.19	69.81	74.53	81.13	75.19	3.48
EBERTconcatAtt-bs8_attclsqw	74.53	79.25	76.42	72.64	82.08	75.47	77.36	67.92	80.19	79.25	76.51	3.92
EBERTconcatAttwithProj-bs8	83.02	80.19	75.47	79.25	83.02	71.70	76.42	80.19	77.36	76.42	78.30	3.35
EBERTconcatAttwithProj-bs8_attclsqw	78.30	81.13	78.30	74.53	83.96	81.13	82.08	82.08	76.42	79.25	79.72	2.74
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery	76.42	71.70	68.87	79.25	80.19	74.53	73.58	77.36	70.75	81.13	75.38	3.97
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	78.30	75.47	81.13	75.47	77.36	73.58	72.64	78.30	76.42	75.47	76.42	2.35
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery	75.47	75.47	77.36	83.96	83.02	76.42	81.13	82.08	74.53	77.36	78.68	3.33
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	76.42	70.75	76.42	81.13	79.25	73.58	79.25	73.58	75.47	79.25	76.51	3.08
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery	76.42	78.30	70.75	70.75	75.47	83.96	76.42	75.47	82.08	79.25	76.89	4.05
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	78.30	75.47	82.08	74.53	72.64	70.75	72.64	78.30	70.75	83.02	75.85	4.20
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery	80.19	76.42	78.30	76.42	75.47	76.42	75.47	75.47	76.42	79.25	76.98	1.59
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	81.13	78.30	74.53	76.42	84.91	77.36	76.42	76.42	77.36	78.30	78.11	2.79

macsay-witai

• Epoch 100 (bs8)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs8	94.34	97.17	94.34	95.28	94.34	95.28	95.28	95.28	93.40	96.23	95.09	1.02
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs8	95.28	95.28	95.28	94.34	97.17	94.34	94.34	93.40	93.40	96.23	94.91	1.13
BERTwithAttClsProjection-bs8	96.23	95.28	92.45	93.40	92.45	96.23	94.34	97.17	94.34	95.28	94.72	1.53
BERTwithProjection-bs8	97.17	96.23	95.28	95.28	95.28	95.28	95.28	94.34	95.28	95.28	95.47	0.71
BERTwithProjectionAtt-bs8	95.28	96.23	96.23	96.23	96.23	94.34	96.23	94.34	94.34	94.34	95.38	0.89
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8	93.40	95.28	91.51	97.17	95.28	95.28	94.34	95.28	95.28	95.28	94.81	1.42
EmbraceBERT-bs8-p_attclsqw	95.28	93.40	95.28	95.28	95.28	94.34	96.23	94.34	97.17	96.23	95.28	1.03
EmbraceBERTwithProj-bs8	96.23	96.23	94.34	94.34	95.28	92.45	95.28	96.23	95.28	97.17	95.28	1.27
EmbraceBERTwithProj-bs8-p_attclsqw	95.28	93.40	95.28	98.11	95.28	94.34	94.34	95.28	94.34	94.34	95.00	1.20
EBERTconcatAtt-bs8	95.28	94.34	95.28	95.28	95.28	96.23	96.23	96.23	94.34	97.17	95.57	0.85
EBERTconcatAtt-bs8_attclsqw	95.28	94.34	96.23	96.23	94.34	91.51	95.28	96.23	94.34	94.34	94.81	1.35
EBERTconcatAttwithProj-bs8	94.34	94.34	96.23	95.28	97.17	94.34	96.23	94.34	88.68	94.34	94.53	2.18
EBERTconcatAttwithProj-bs8_attclsqw	91.51	94.34	94.34	96.23	93.40	91.51	96.23	94.34	95.28	95.28	94.25	1.60
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery	94.34	94.34	95.28	95.28	88.68	94.34	94.34	96.23	93.40	92.45	93.87	1.99
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	93.40	93.40	93.40	93.40	95.28	93.40	94.34	96.23	95.28	94.34	94.25	0.98
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery	96.23	95.28	95.28	96.23	94.34	94.34	96.23	94.34	96.23	96.23	95.47	0.82
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	96.23	96.23	95.28	95.28	93.40	95.28	94.34	94.34	95.28	90.57	94.62	1.58
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery	95.28	97.17	95.28	94.34	96.23	92.45	95.28	95.28	94.34	93.40	94.91	1.28
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	96.23	95.28	94.34	93.40	95.28	94.34	94.34	94.34	92.45	95.28	94.53	1.02
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery	97.17	95.28	93.40	96.23	96.23	95.28	94.34	95.28	95.28	97.17	95.57	1.12
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	94.34	94.34	95.28	95.28	95.28	96.23	95.28	96.23	95.28	96.23	95.38	0.66