-
Notifications
You must be signed in to change notification settings - Fork 35
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Test result does not match the paper #3
Comments
hi, I think there is something wrong with your feature test code. I have tested this model and the result can achieve 69.5 Recall@1 |
@dusigh Please refer the test code here and check the possible errors . https://github.com/PkuRainBow/Hard-Aware-Deeply-Cascaded-Embedding_release/blob/master/src_code/test_stanford_products.py |
Could you update the data/stanford_products/test_label.dat file in the repository for the stanford_product dataset so that I can make our experiments as comparable as possible? The current one in this repository obviously has some problems: The labels are simply the sequential index: "0, 1, 2, 3, 4 ... 60498". |
@dusigh Hope this file could help you |
Thank you. Yes, this version works. |
I checked out your code and ran test on the StanfordOnlineProduct dataset. I used your pre-trained model: StanfordProducts_FastPair_cascade_v1_hard_ratio_iter_60000_50_25_10_672.caffemodel.
The test result is as follows:
stanford online products mean recall@ 1 : 0.450450
stanford online products mean recall@ 10 : 0.685800
stanford online products mean recall@ 100 : 0.785500
stanford online products mean recall@ 1000 : 0.860150
This doesn't seem to match the result in the paper:
HDC + Contrastive 384 69.5 (@1) 84.4(@10) 92.8(@100) 97.7(@1000)
I do find out that the labels in your HDC_train.txt file differ from those in Ebay_train.txt by 1. For example: mug_final/151771651316_1.JPG has label 6687 in Evay_train.txt but has label 6686 in HDC_train.txt. Anyway I don't think that matters because as long as the labels are consistent within the same file it should be fine, not to mention that I am using the pre-trained model so those files were not used at all.
Could you give me some hint on what might have gone wrong so that the test results do not match the paper? Thank you!
The text was updated successfully, but these errors were encountered: