-
Notifications
You must be signed in to change notification settings - Fork 54
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
why the best score can't get 89.2 when I use quora.json5 to train re2? #1
Comments
You should set "eval_file" to "test" (like what "main.json5" does), which evaluates the test set directly in each evaluation during training. This is not what we should do in production, but aligns with previous works and makes the results comparable. |
I use "main.json5", and the acc of quora test set can't get 89.2%, only about 84%. |
Did you download the Quora dataset from the link in ReadMe and prepare the data using "prepare_quora.py"? |
yes~ |
I did. |
Then that's strange... What's your pytorch version? I'll rerun the experiment later |
my pytorch version is 1.1.0 . |
Hi ,I was testing RE2 (Pytorch 1.3.1) on quora dataset, and I got results that were aligned with the paper :
|
I'm trying to reproduce the result on Pytorch 1.5.1 and can confirm that the masking issue is gone with the above fix. I, however, ran into another issue, as below:
I'm fairly new to Pytorch and still learning the basics but it does seem like 'model.eval()' was changed somehow, in the newer version. Has anyone encountered this issue before and is there a workaround? Thanks in advance! |
@TheMnBN 'gnorm': grad_norm.item(), should solve the problem. |
Hey, Very Nice Work. I started training this model and it got failed after few hours giving out of memory error.
File "/content/simple-effective-text-matching-pytorch/src/modules/fusion.py", line 50, in forward If possible can anyone please provide me the pre-trained model for inference directly? Thanks in advance!! |
I use quora.json5 to train a new re2, and use the new model to test. But the acc of quora test set can't get 89.2%.
The text was updated successfully, but these errors were encountered: