low accuracy on megaface with pretrained model

[qunluo]

Hi, I download pretrained model(model-r50-am-lfw.zip) from https://pan.baidu.com/s/1mj6X7MK.
I follow readme to test model-0000.params on megaface. The accuracy is very low.
Is there something wrong with model-0000.params?

result is as follow:
1.000000
Done matching! Score matrix size: 3526 1000000
Saving to ./sphereface_results/otherFiles/facescrub.json_megaface_mxsphereface20c_112x112_1000000_1.bin
Computing test results with 1000000 images for set 1
Loaded 3526 probes spanning 80 classes
Loading from ./sphereface_results/otherFiles/facescrub.json_facescrub.json_mxsphereface20c_112x112.bin
Probe score matrix size: 3526 3526
distractor score matrix size: 3526 1000000
Done loading. Time to compute some stats!
Finding top distractors!
Done sorting distractor scores
Making gallery!
Done Making Gallery!
Allocating ranks (1000080)
Rank 1: 0.020466

[nttstar]

First check your input, do not subtraction mean by using our model.

[qunluo]

thanks. I will try again without subtracting mean.
without subtracting mean, I can get rank 1: 0.798349. Is this normal?(I didn't remove noise)

[nttstar]

Yes, a normal result. r50-lfw pretrained model was trained with a smaller training dataset which removed many candidate lfw-overlapped identities.

[qunluo]

After running remove_noises.py, I got the result Rank 1: 0.966336.

[lucwan]

@nttstar paper reports r50 accuracy as Rank 1: 82.42 in Table 10 when it is trained with no overlap with facescrub dataset. So the accuracy decreased to 79.8 from 82.42 when lfw-overlapped identities are removed?

[nttstar]

We got 82.42 after removing facescrub noises. Also the accuracy before removing megaface noises is not stable, you may need N runs to pick the best.
Another note is we removed 5000+ identities from training set to prevent overlapping with LFW, a lot of images.

[lucwan]

The accuracy after using remove_noise.py is 96.6% whereas the actual accuracy reported on github page is 97.6% for r50. What could be the reason for this difference?

[nttstar]

@qunluo Did you use the aligned facescrub images we provided?

[nttstar]

@lucwan I checked my page again and it seems we're using the same model to test MegaFace, so the problem may caused by aligned facescrub and megaface distractor images.

[qunluo]

@nttstar I didn't. I use align_facescrub.py and align_megaface.py to align image.

[nttstar]

@qunluo It may have some differences. Use the provided aligned facescrub images if you have time to test it again, thanks~

[qunluo]

@nttstar I am downloading the data. I am closing the issue now and will reopen it if necessary.

[qunluo]

Using the provided aligned facescrub images can get Rank 1: 0.976174.

[nttstar]

It is now consistent, thank you.