Implement the LAMBADA evaluation

[StellaAthena]

The LAMBADA dataset [PKL+16] tests the modeling of long-range dependencies in text – the model is asked to predict the last word of sentences which require reading a paragraph of context. It has recently been suggested that the continued scaling of language models is yielding diminishing returns on this difficult benchmark. [BHT+20] reflect on the small 1.5% improvement achieved by a doubling of model size between two recent state of the art results ([SPP+19]and [Tur20]) and argue that "continuing to expand hardware and data sizes by orders of magnitude is not the path forward”. We find that path is still promising and in a zero-shot setting GPT-3 achieves 76% on LAMBADA, a gain of 8% over the previous state of the art.

[cfoster0]

Original dataset is available at the link below:

https://zenodo.org/record/2630551

[sdtblck]

this is the dataset openAI used for evaluating GPT2 #39
(reference: openai/gpt-2#131)

[leogao2]

@anishthite is this done/are you still working on this?

[leogao2]

I think I'll be taking over this one