The transformative power of transformers in protein structure prediction

by Bernard Moussad, Rahmatullah Roche, Debswapna Bhattacharya

published in PNAS

This repo contains all PDBs generated and metrics calculated to benchmark the predictive modeling performance of AlphaFold2, RoseTTAFold, ESMFold, and OmegaFold on a set of 69 CASP15 single-chain protein targets with available experimental coordinates.

The predicted structures for each method are in their respective folders. For MSA-based methods, the link to the MSAs are provided.

The metrics used are:

• Global Distance Test - Total Score (GDT-TS)
• Global Distance Calculation - Side-chain (GDC-SC)
• Template Modeling Score (TMscore)
• Local Distance Difference Test (LDDT)
• MolProbity score (MolProbity)

GDT-TS and GDC-SC are part of the Local Global Alignment (LGA) tool which can be download here

LDDT is a standalone tool which can be downloaded here

TM-score is also available for download here

MolProbity can be installed locally via the GitHub Repositry here or run via the webserver here

1. AlphaFold2 Results
2. RoseTTAFold Results
3. ESMFold Results
4. OmegaFold Results

AlphaFold2

MSAs located at AlphaFold2 MSAs

Target	GDT-TS	GDC-SC	TMscore	Global LDDT	MolProbity
T1104.pdb	94.02	61.32	0.95	0.86	1.07
T1106s1.pdb	72.18	30.15	0.66	0.77	1.91
T1106s2.pdb	92.34	56.21	0.93	0.89	1.12
T1109.pdb	87.67	63.17	0.89	0.88	0.93
T1110.pdb	95.82	74.86	0.97	0.92	0.87
T1112.pdb	76.69	38.67	0.88	0.83	0.84
T1113.pdb	66.47	43.42	0.68	0.83	1.49
T1114s1.pdb	40.15	14.99	0.43	0.83	1.52
T1114s2.pdb	94.10	67.91	0.98	0.87	1.34
T1114s3.pdb	96.54	71.71	0.99	0.90	1.21
T1119.pdb	92.19	52.68	0.85	0.82	1.35
T1120.pdb	50.66	28.87	0.52	0.83	0.72
T1121.pdb	51.03	25.28	0.60	0.80	0.92
T1122.pdb	26.20	7.53	0.37	0.41	2.10
T1123.pdb	74.18	39.35	0.83	0.66	2.15
T1124.pdb	89.88	62.83	0.95	0.87	1.08
T1125.pdb	12.23	4.16	0.26	0.30	1.74
T1127.pdb	88.17	56.47	0.95	0.87	0.97
T1129s2.pdb	63.50	37.24	0.82	0.56	1.59
T1130.pdb	36.01	9.79	0.37	0.45	1.82
T1131.pdb	20.03	5.21	0.23	0.29	2.12
T1132.pdb	97.45	73.39	0.96	0.90	0.71
T1133.pdb	93.95	68.99	0.98	0.90	1.53
T1134s1.pdb	93.12	69.30	0.97	0.91	0.92
T1134s2.pdb	85.02	59.72	0.89	0.84	0.92
T1137s1.pdb	36.67	16.74	0.42	0.80	0.85
T1137s2.pdb	37.90	21.72	0.42	0.85	0.89
T1137s3.pdb	35.22	15.95	0.40	0.83	1.19
T1137s4.pdb	28.00	13.69	0.31	0.81	1.07
T1137s5.pdb	40.01	22.11	0.49	0.83	0.83
T1137s6.pdb	36.15	17.07	0.46	0.83	1.20
T1137s7.pdb	87.67	54.55	0.96	0.85	1.60
T1137s8.pdb	92.43	56.81	0.97	0.86	0.60
T1137s9.pdb	90.09	57.23	0.97	0.88	0.82
T1139.pdb	87.03	56.83	0.96	0.85	1.13
T1145.pdb	69.74	37.20	0.82	0.88	1.12
T1146.pdb	94.44	65.47	0.97	0.86	1.48
T1147.pdb	98.64	71.79	0.98	0.91	1.16
T1150.pdb	94.06	65.15	0.98	0.88	1.17
T1151s2.pdb	91.37	54.41	0.90	0.81	1.34
T1152.pdb	96.19	59.82	0.88	0.83	1.67
T1153.pdb	92.78	72.67	0.95	0.88	1.09
T1154.pdb	24.30	11.24	0.41	0.51	1.49
T1155.pdb	65.63	22.13	0.65	0.58	1.58
T1157s1.pdb	57.54	35.23	0.70	0.67	1.28
T1157s2.pdb	47.31	15.46	0.66	0.78	1.36
T1158.pdb	61.06	30.84	0.90	0.84	1.28
T1159.pdb	97.81	71.34	0.98	0.93	1.01
T1160.pdb	52.59	5.71	0.27	0.62	0.80
T1161.pdb	53.64	21.38	0.45	0.58	0.80
T1162.pdb	94.64	68.64	0.97	0.89	1.19
T1163.pdb	94.43	67.22	0.96	0.90	0.88
T1170.pdb	69.15	39.65	0.81	0.85	0.84
T1173.pdb	44.24	17.70	0.52	0.60	2.03
T1174.pdb	40.98	23.29	0.53	0.74	1.46
T1175.pdb	93.37	60.36	0.97	0.85	1.29
T1176.pdb	73.53	40.98	0.77	0.78	0.63
T1177.pdb	90.13	57.70	0.96	0.89	0.98
T1178.pdb	75.09	41.41	0.85	0.70	1.94
T1179.pdb	78.28	45.68	0.87	0.70	1.71
T1180.pdb	87.14	52.99	0.95	0.88	0.84
T1181.pdb	69.04	41.54	0.82	0.84	1.32
T1182.pdb	89.36	60.65	0.98	0.91	1.27
T1183.pdb	97.05	70.22	0.98	0.91	0.85
T1184.pdb	97.60	72.00	0.98	0.91	1.82
T1186.pdb	99.90	85.18	1.00	0.97	0.67
T1187.pdb	93.44	60.92	0.94	0.87	0.91
T1188.pdb	97.12	72.41	0.99	0.93	1.27
T1194.pdb	98.91	64.15	0.99	0.90	1.20
Mean	73.06	45.48	0.78	0.80	1.23

RoseTTAFold

MSAs located at RoseTTAFold MSAs

Target	GDT-TS	GDC-SC	TMscore	Global LDDT	MolProbity
T1104.pdb	53.63	13.46	0.59	0.53	1.07
T1106s1.pdb	67.96	26.80	0.63	0.73	1.32
T1106s2.pdb	86.26	42.71	0.89	0.82	1.30
T1109.pdb	82.69	49.43	0.87	0.79	1.51
T1110.pdb	88.24	54.50	0.92	0.81	1.16
T1112.pdb	53.75	18.24	0.73	0.71	1.37
T1113.pdb	30.39	9.88	0.36	0.40	1.91
T1114s1.pdb	23.38	7.59	0.24	0.72	1.52
T1114s2.pdb	77.25	39.23	0.90	0.70	1.44
T1114s3.pdb	86.75	47.85	0.97	0.74	1.47
T1119.pdb	65.10	25.27	0.57	0.68	1.73
T1120.pdb	40.57	16.61	0.49	0.62	1.76
T1121.pdb	37.53	16.21	0.52	0.63	1.37
T1122.pdb	18.04	3.92	0.26	0.34	2.13
T1123.pdb	41.94	11.19	0.57	0.36	1.92
T1124.pdb	53.57	28.10	0.63	0.70	1.35
T1125.pdb	6.33	1.86	0.18	0.24	1.93
T1127.pdb	80.49	38.69	0.88	0.77	1.17
T1129s2.pdb	17.01	6.15	0.34	0.25	1.95
T1130.pdb	23.43	3.13	0.30	0.38	1.74
T1131.pdb	14.91	1.21	0.19	0.26	2.03
T1132.pdb	93.62	52.66	0.94	0.80	1.34
T1133.pdb	75.92	41.51	0.93	0.77	1.48
T1134s1.pdb	76.31	38.33	0.88	0.80	0.69
T1134s2.pdb	75.09	40.91	0.86	0.75	1.46
T1137s1.pdb	29.10	8.42	0.35	0.68	1.32
T1137s2.pdb	30.39	13.58	0.33	0.76	1.02
T1137s3.pdb	33.14	15.02	0.38	0.76	1.39
T1137s4.pdb	24.94	11.72	0.29	0.65	1.46
T1137s5.pdb	31.92	15.84	0.37	0.70	1.35
T1137s6.pdb	27.57	11.22	0.32	0.70	1.27
T1137s7.pdb	76.42	40.56	0.92	0.73	1.52
T1137s8.pdb	82.57	43.53	0.91	0.78	1.22
T1137s9.pdb	76.70	37.92	0.90	0.77	1.07
T1139.pdb	55.25	27.32	0.69	0.64	1.85
T1145.pdb	31.53	12.19	0.48	0.56	1.75
T1146.pdb	82.80	40.80	0.94	0.73	1.65
T1147.pdb	90.22	52.63	0.91	0.78	1.29
T1150.pdb	79.59	41.56	0.94	0.73	1.36
T1151s2.pdb	78.57	32.48	0.78	0.70	1.33
T1152.pdb	97.28	55.27	0.91	0.81	0.50
T1153.pdb	82.03	46.95	0.90	0.74	1.49
T1154.pdb	7.11	2.67	0.16	0.20	2.30
T1155.pdb	60.82	19.41	0.64	0.53	1.70
T1157s1.pdb	45.64	22.24	0.62	0.57	1.29
T1157s2.pdb	40.20	11.83	0.57	0.73	1.12
T1158.pdb	42.66	18.84	0.71	0.74	1.38
T1159.pdb	87.50	51.50	0.94	0.80	1.82
T1160.pdb	53.45	2.25	0.27	0.59	2.05
T1161.pdb	58.33	15.28	0.42	0.57	1.22
T1162.pdb	74.56	34.08	0.85	0.64	1.44
T1163.pdb	76.36	41.62	0.85	0.74	1.47
T1170.pdb	58.57	26.41	0.75	0.78	0.91
T1173.pdb	24.51	9.43	0.27	0.34	1.68
T1174.pdb	13.76	2.72	0.21	0.37	1.78
T1175.pdb	66.19	27.55	0.82	0.64	1.55
T1176.pdb	60.73	20.86	0.68	0.65	1.72
T1177.pdb	64.01	19.40	0.72	0.63	1.26
T1178.pdb	50.09	15.46	0.69	0.47	1.84
T1179.pdb	36.71	13.33	0.51	0.35	2.26
T1180.pdb	56.95	33.89	0.63	0.75	1.16
T1181.pdb	43.83	19.28	0.63	0.56	1.82
T1182.pdb	65.46	26.96	0.89	0.70	1.50
T1183.pdb	76.54	35.69	0.87	0.67	1.52
T1184.pdb	54.60	32.07	0.59	0.51	1.28
T1186.pdb	95.17	61.94	0.98	0.86	1.33
T1187.pdb	71.80	30.60	0.80	0.66	1.56
T1188.pdb	62.78	33.20	0.80	0.68	1.73
T1194.pdb	63.66	22.78	0.76	0.62	1.45
Mean	56.41	26.00	0.65	0.64	1.49

ESMFold

Target	GDT-TS	GDC-SC	TMscore	Global LDDT	MolProbity
T1104.pdb	82.69	41.80	0.87	0.72	3.07
T1106s1.pdb	49.65	19.03	0.46	0.67	3.59
T1106s2.pdb	94.37	62.11	0.94	0.90	1.72
T1109.pdb	87.56	60.77	0.89	0.88	1.94
T1110.pdb	95.25	69.25	0.97	0.91	1.91
T1112.pdb	66.74	31.58	0.86	0.80	2.23
T1113.pdb	21.56	7.27	0.26	0.36	3.79
T1114s1.pdb	35.15	14.81	0.39	0.82	2.45
T1114s2.pdb	69.18	42.60	0.79	0.68	2.96
T1114s3.pdb	89.23	60.43	0.96	0.80	2.54
T1119.pdb	98.44	58.59	0.94	0.86	1.36
T1120.pdb	52.19	23.17	0.53	0.78	2.16
T1121.pdb	52.53	28.15	0.62	0.80	2.01
T1122.pdb	36.74	16.22	0.45	0.47	3.69
T1123.pdb	20.09	6.88	0.27	0.20	3.99
T1124.pdb	79.90	50.17	0.90	0.81	2.23
T1125.pdb	14.86	6.73	0.29	0.34	3.90
T1127.pdb	88.78	57.48	0.93	0.82	2.45
T1129s2.pdb	6.21	1.60	0.19	0.15	4.23
T1130.pdb	29.24	5.61	0.36	0.40	3.69
T1131.pdb	17.70	4.07	0.22	0.27	3.71
T1132.pdb	94.39	62.02	0.92	0.84	1.96
T1133.pdb	93.03	67.97	0.98	0.89	2.49
T1134s1.pdb	89.63	62.66	0.95	0.86	1.94
T1134s2.pdb	83.49	56.53	0.88	0.83	2.71
T1137s1.pdb	29.74	12.73	0.35	0.69	3.27
T1137s2.pdb	33.89	15.55	0.36	0.80	2.86
T1137s3.pdb	34.83	16.67	0.38	0.82	3.41
T1137s4.pdb	29.77	15.63	0.33	0.65	3.53
T1137s5.pdb	31.36	18.27	0.35	0.72	3.13
T1137s6.pdb	30.58	11.63	0.36	0.71	3.42
T1137s7.pdb	77.00	47.06	0.84	0.74	3.55
T1137s8.pdb	90.44	56.92	0.95	0.85	1.44
T1137s9.pdb	86.70	52.53	0.95	0.84	2.04
T1139.pdb	56.44	34.22	0.62	0.59	2.81
T1145.pdb	29.73	15.10	0.44	0.54	3.40
T1146.pdb	87.45	56.21	0.93	0.81	2.46
T1147.pdb	94.30	59.69	0.94	0.86	2.67
T1150.pdb	92.98	60.67	0.96	0.83	2.21
T1151s2.pdb	86.61	48.49	0.84	0.76	3.25
T1152.pdb	98.37	66.59	0.93	0.89	2.66
T1153.pdb	83.94	59.28	0.90	0.79	2.66
T1154.pdb	7.11	3.62	0.18	0.20	4.40
T1155.pdb	68.75	27.89	0.71	0.61	2.82
T1157s1.pdb	47.92	28.69	0.61	0.56	3.45
T1157s2.pdb	42.78	14.61	0.62	0.78	2.19
T1158.pdb	52.91	24.06	0.80	0.81	2.53
T1159.pdb	74.38	38.45	0.80	0.71	3.04
T1160.pdb	53.45	8.40	0.29	0.62	2.33
T1161.pdb	63.54	35.14	0.57	0.60	2.11
T1162.pdb	77.53	45.92	0.83	0.69	3.39
T1163.pdb	82.61	46.47	0.87	0.79	2.45
T1170.pdb	67.71	39.00	0.80	0.84	1.76
T1173.pdb	32.84	17.26	0.36	0.37	3.47
T1174.pdb	44.30	23.01	0.55	0.72	3.05
T1175.pdb	88.64	52.79	0.93	0.81	2.48
T1176.pdb	72.50	39.57	0.77	0.78	2.53
T1177.pdb	73.54	40.44	0.79	0.79	1.88
T1178.pdb	9.44	1.79	0.18	0.17	3.96
T1179.pdb	10.91	3.51	0.18	0.17	4.01
T1180.pdb	60.50	40.40	0.64	0.82	1.72
T1181.pdb	53.49	30.13	0.68	0.69	3.21
T1182.pdb	72.23	42.88	0.90	0.71	2.86
T1183.pdb	96.03	67.04	0.97	0.89	2.10
T1184.pdb	24.00	3.10	0.27	0.26	3.76
T1186.pdb	100.00	81.04	1.00	0.95	1.54
T1187.pdb	90.24	57.71	0.93	0.86	2.50
T1188.pdb	66.80	47.82	0.73	0.75	3.21
T1194.pdb	94.56	53.08	0.95	0.82	2.37
Mean	61.62	35.92	0.67	0.69	2.79

OmegaFold

Target	GDT-TS	GDC-SC	TMscore	Global LDDT	MolProbity
T1104.pdb	45.73	11.18	0.46	0.43	3.40
T1106s1.pdb	49.65	16.04	0.47	0.67	3.15
T1106s2.pdb	90.77	56.52	0.91	0.85	1.61
T1109.pdb	87.67	62.10	0.89	0.88	2.04
T1110.pdb	95.36	71.08	0.97	0.91	2.09
T1112.pdb	70.44	35.68	0.89	0.80	2.22
T1113.pdb	22.60	4.99	0.29	0.40	3.72
T1114s1.pdb	47.20	19.05	0.49	0.83	2.38
T1114s2.pdb	55.59	34.12	0.65	0.58	3.40
T1114s3.pdb	73.29	42.49	0.89	0.67	2.78
T1119.pdb	99.48	62.73	0.95	0.88	1.42
T1120.pdb	45.62	24.48	0.50	0.78	2.24
T1121.pdb	47.74	22.52	0.54	0.77	1.88
T1122.pdb	27.83	7.18	0.37	0.42	3.64
T1123.pdb	17.29	6.65	0.25	0.22	3.90
T1124.pdb	74.14	39.99	0.87	0.79	2.45
T1125.pdb	11.37	3.48	0.19	0.29	3.33
T1127.pdb	90.12	57.76	0.93	0.81	2.56
T1129s2.pdb	12.39	6.23	0.25	0.17	3.63
T1130.pdb	30.03	8.10	0.37	0.39	3.68
T1131.pdb	20.65	2.52	0.27	0.31	3.50
T1132.pdb	86.73	44.61	0.86	0.76	1.87
T1133.pdb	91.75	66.82	0.98	0.88	2.29
T1134s1.pdb	90.17	62.22	0.95	0.86	1.86
T1134s2.pdb	85.38	61.94	0.89	0.85	2.60
T1137s1.pdb	33.46	16.26	0.37	0.69	3.35
T1137s2.pdb	32.73	15.03	0.35	0.80	2.75
T1137s3.pdb	33.15	13.78	0.35	0.81	2.90
T1137s4.pdb	29.82	12.61	0.34	0.66	3.10
T1137s5.pdb	32.82	18.22	0.37	0.71	3.20
T1137s6.pdb	29.38	13.79	0.33	0.67	3.14
T1137s7.pdb	83.00	47.20	0.92	0.77	3.17
T1137s8.pdb	89.64	56.28	0.95	0.86	2.14
T1137s9.pdb	76.60	41.33	0.90	0.82	2.24
T1139.pdb	56.70	35.02	0.63	0.58	2.95
T1145.pdb	19.47	9.42	0.34	0.43	3.32
T1146.pdb	76.25	41.10	0.87	0.72	3.14
T1147.pdb	94.02	59.86	0.92	0.85	2.33
T1150.pdb	71.27	34.73	0.87	0.72	2.77
T1151s2.pdb	87.80	46.69	0.87	0.76	3.20
T1152.pdb	94.02	58.09	0.86	0.83	3.08
T1153.pdb	81.37	57.68	0.88	0.77	3.27
T1154.pdb	7.07	3.88	0.13	0.18	3.58
T1155.pdb	66.83	27.33	0.70	0.58	3.19
T1157s1.pdb	46.08	28.55	0.54	0.53	3.20
T1157s2.pdb	47.58	18.74	0.63	0.77	1.69
T1158.pdb	58.52	29.49	0.83	0.82	2.61
T1159.pdb	58.75	27.10	0.67	0.59	3.37
T1160.pdb	52.59	7.36	0.27	0.62	2.34
T1161.pdb	52.61	18.58	0.44	0.56	2.29
T1162.pdb	70.24	40.27	0.74	0.63	3.42
T1163.pdb	82.20	45.06	0.87	0.76	2.66
T1170.pdb	68.99	40.03	0.81	0.85	1.75
T1173.pdb	27.57	13.93	0.33	0.31	3.65
T1174.pdb	43.34	19.85	0.57	0.68	2.98
T1175.pdb	92.14	56.52	0.96	0.83	2.58
T1176.pdb	68.09	30.20	0.71	0.72	2.52
T1177.pdb	73.99	42.90	0.80	0.78	2.28
T1178.pdb	8.24	0.67	0.15	0.16	3.80
T1179.pdb	9.72	3.44	0.17	0.16	3.93
T1180.pdb	59.32	40.92	0.64	0.82	1.80
T1181.pdb	49.74	28.83	0.63	0.66	3.36
T1182.pdb	23.42	12.89	0.34	0.34	3.58
T1183.pdb	82.44	53.01	0.87	0.76	2.55
T1184.pdb	35.60	9.19	0.39	0.36	3.69
T1186.pdb	97.88	78.59	0.98	0.93	1.62
T1187.pdb	87.50	56.61	0.91	0.84	2.48
T1188.pdb	65.53	45.30	0.72	0.73	3.12
T1194.pdb	85.41	44.39	0.90	0.72	2.77
Mean	58.14	32.34	0.64	0.66	2.82