Reproduce performance of structshot

[nguyennth]

Hi,

Thank you for publishing the source code. I have access to OntoNotes and try to reproduce the results in your paper.

In the case of 1-shot learning with NNshot, I could obtain the same average score, i.e., 33.2, although the scores of each dataset are different from yours. However, I'm struggling with Structshot. My scores on CoNLL is much lower than yours, resulting in a lower average score on the three datasets.

In the case of 5-shot learning, I couldn't obtain as high scores as yours both with NNshot and Structshot. In average, Structshot produced worse performance than NNshot.

I wonder if you use a different set of parameters than the one you publish here. I hope you can help me to shed some light here.

Thank you very much for your time!

Best regards,
Nhung

[yi-asapp]

Hi Nhung,

Thanks for your interest in our work. For OntoNotes, did you use the splits noted here: https://cemantix.org/conll/2012/data.html ?

The most important parameter for StructShot is tau, which you can tune here: https://github.com/asappresearch/structshot/blob/main/structshot/run_pred.sh#L22

It is a sensitive one, so I would suggest to try values like [0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1]

[nguyennth]

Hi Yi,

Thank you very much for your prompt responses.

I use the splits mentioned here: https://cemantix.org/data/ontonotes.html, which I think is probably the same as the splits that you shared. Anyway, I'll double-check them.

Thanks for your suggestions of tau values!