beam search ranked by ppl instead of probability

[lichengunc]

The code is trained on minimizing log_perplexity, but the beam search is ranked by log_probs.
A simple test on my side shows ranking by log_perplexity in beam search could give higher bleu, rouge, and meteor scores, which is more consistent with the optimization function.

[karpathy]

doh! Sorry, can you elaborate? I was under the impression that logprobs was already normalized due to use of a LogSoftMax layer, so this should already be a correctly-normalized log-perplexity? What change did you include, exactly?

[lichengunc]

Hi, the nn.LanguageModelCriterion is optimized by minimizing -logprobs/#total_number_wds within a batch. I would consider it as log_ppl. However, during beam search, we are choosing top K beams with highest logprobs:
L169: local function compare(a,b) return a.p > b.p end -- used downstream
I think considering ppl and "return a.ppl < b.ppl" is more reasonable.
How do you think?

[karpathy]

When I create the things I end up sorting, I create them on L218 as

table.insert(candidates, {c=ix[{ q,c }], q=q, p=candidate_logprob, r=local_logprob })

so in fact the .p field holds the logprob, which I end up sorting by. I'm not using the raw probabilities. And there is no .ppl field here.

[lichengunc]

You are absolutely right here! But I think the ranking of done_beams need to consider logppls.
What I did is add one more function called "compare_ppl" and I will calculate ppl for each done_beam, so that

will rank done_beams by ascending logppls, instead of descending logprobs. How do you think?

[szq0214]

@lichengunc I think you're right. I also achieve higher performance after sorting with ppl.