Current pretrained models test performance does not match reported

[swabhs]

There might be an issue with the way the models are being saved, I am currently looking into this.

[silentrob]

I'm also seeing unexpected frames. Given, I would expect a GIFT frame.

1	jeff	_	UNK	_	NNP	0	_	_	_	_	_	_	_	O
2	gave	_	give	_	VBD	0	_	_	_	_	_	_	_	O
3	the	_	the	_	DT	0	_	_	_	_	_	_	_	O
4	book	_	book	_	NN	0	_	_	_	_	_	_	_	O
5	to	_	to	_	TO	0	_	_	_	_	_	_	_	O
6	rob	_	Rob	_	NNP	0	_	_	_	_	_	rob.v	Robbery	O
7	after	_	after	_	IN	0	_	_	_	_	_	_	_	O
8	class	_	class	_	NN	0	_	_	_	_	_	_	_	O
9	.	_	.	_	.	0	_	_	_	_	_	_	_	O

1	jeff	_	UNK	_	NNP	0	_	_	_	_	_	_	_	O
2	gave	_	give	_	VBD	0	_	_	_	_	_	_	_	O
3	the	_	the	_	DT	0	_	_	_	_	_	_	_	O
4	book	_	book	_	NN	0	_	_	_	_	_	_	_	O
5	to	_	to	_	TO	0	_	_	_	_	_	_	_	O
6	rob	_	Rob	_	NNP	0	_	_	_	_	_	_	_	O
7	after	_	after	_	IN	0	_	_	_	_	_	after.prep	Time_vector	O
8	class	_	class	_	NN	0	_	_	_	_	_	_	_	O
9	.	_	.	_	.	0	_	_	_	_	_	_	_	O

[swabhs]

The current status of this problem seems to be this - models trained on one machine perform as expected on the same machine, but change behavior on other machines. I was using different images of the same machine and was able to use pretrained models to replicate the results I have reported. However, if I use another machine I'm not able to replicate the same results.

Currently I have no idea why this might be happening - my suspicion is it might have something to do with DyNet, since there is nothing particularly different I'm doing with saving and loading models. Any help or suggestions are most welcome!

The fix for now is to train your own models. I have enough evidence now to believe that retraining these models results in performances similar to what I have reported.