Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

<Fix> evaluation dataset, printed samples #836

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

hypothe
Copy link

@hypothe hypothe commented Jan 5, 2023

Bunch of minor "theoretical" changes in the evaluation function:

  1. test_data_gen was used as the data generator in the evaluation, instead of data_generator, thereby evaluating the net on the test set used for training (not an actual issue here given the sequences are randomized and not sampled from existing datasets, but in principle would lead to a data leak in realistic scenarios);
  2. the correct sequences printed were a sampling (with reinsertion) of the first 10 evaluated, instead of 10 sampled from the whole set of correct ones;
  3. the condition for printing the incorrectly classified sequences would declare the absence of misclassifications if verbose==False, independently of their actual presence;

@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@@ -497,7 +497,7 @@
"\n",
" with torch.no_grad():\n",
" for batch_idx in range(len(data_generator)):\n",
" data, target = test_data_gen[batch_idx]\n",
" data, target = data_generator[batch_idx]\n",
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We want to inspect sequences from the test set, no?
I didn't get it.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then why are we declaring the "data_generator", with a different seed, and not using it to sample data?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants