Different Recall&Accuracy Results

[anil1055]

Hi,

I ran your code successfully and some results are different, especially for batch_size: 48. I used your parameters, but accuracy and recall values' differences are overmuch. I wonder why, maybe you changed other parameters. Especially for batch size: 48, why accuracies and recalls values are quite different according to your results.

Also biobert model didn't run, I got an error

Have a nice day.

===PARAMETERS===
debug False
debug_data_num 200
dataset bc5cdr
dataset_dir ./dataset/
serialization_dir ./serialization_dir/
preprocessed_doc_dir ./preprocessed_doc_dir/
kb_dir ./mesh/
cached_instance False
lr 1e-05
weight_decay 0
beta1 0.9
beta2 0.999
epsilon 1e-08
amsgrad False
word_embedding_dropout 0.1
cuda_devices 0
scoring_function_for_model indexflatip
num_epochs 10
patience 10
batch_size_for_train 48
batch_size_for_eval 48 or 16, I tried this
bert_name bert-base-uncased
max_context_len 50
max_mention_len 12
max_canonical_len 12
max_def_len 36
model_for_training biencoder I tried other models
candidates_dataset ./candidates.pkl
max_candidates_num 10
search_method_for_faiss indexflatip
how_many_top_hits_preserved 50
===PARAMETERS END===

BioBERT error:

[izuna385]

I'm sorry for the delay in replying, and thank you for the detailed comparison experiment.
I don't have time to adjust the parameters for this code. I'll let you know what I noticed about your comment.

• Batch size during training is important, because in-batch training, as used in the BLINK model and Gillick et al's model, batch size matters a lot.

  

• The reason why the biobert model does not load is not immediately clear: if you have a newer version of transformers or allennlp, it is likely that it will not load with this code.