-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SAINT on Ednet data #4
Comments
It would be great for me to have entire codes for reproducing the results on the paper either. |
Hi kwonmha, This implementation of SAINT is not completely finished. All the rest is correct :) I reach AUC=0.76 with it but I'm not able to get the last 2% and also my metrics crash if I use a dimenson_model of 512 like in the paper (it works only with a smaller model) |
Hi, @Nino-SEGALA In my case, I think the problem exists in data processing or data itself, not in modeling. Do you have any plan to upload your code on your github? |
I will try to upload it here with a Pull Request :) I don't understand, it works with EdNet from Kaggle, but not with EdNet from the paper? |
@Nino-SEGALA Here's the link to the dataset I mentioned. |
Yes, I also use this one (and get 0.76 AUC with dim_model=128, if I use a larger model dim_model=512 I get AUC=0.5 me too :/) Maybe you can try with a smaller model And this dataset 'my model works fine with Ednet data from Kaggle' ? :) |
Thanks for informing! |
@Nino-SEGALA Have you tried applying Noam scheme learning rate scheduling mentioned on the paper? I got the same problem where auc stays around 0.5 with dimension 256, 512. Noam scheduler code link
In the class for convenience. As it changes leraning rate regard to step, batch_size looks important which have effect on the number of steps in training. I got 0.7746 AUC with dim 256, 7727 with dim 512 |
Thanks a lot for your comment kwonmha! I did my training without Noam Scheme, and since I have implemented it, I didn't retry to do the big trainings. @kwonmha could you also share your ACC, RMSE and BCE loss if you have them? |
Sorry but I haven't measure metrics other than AUC so far. |
I got 0.7666 AUC with dim 256, 0.7537 with dim 512 |
Would you be able to add your code for running your implementation of SAINT on EdNet as well - besides the example on random data?
The text was updated successfully, but these errors were encountered: