Add bAbI QA tasks

[gchhablani]

• Name: The (20) QA bAbI tasks
• Description: The (20) QA bAbI tasks are a set of proxy tasks that evaluate reading comprehension via question answering. Our tasks measure understanding in several ways: whether a system is able to answer questions via chaining facts, simple induction, deduction and many more. The tasks are designed to be prerequisites for any system that aims to be capable of conversing with a human. The aim is to classify these tasks into skill sets,so that researchers can identify (and then rectify) the failings of their systems.
• Paper: arXiv
• Data: Facebook Research Page
• Motivation: This is a unique dataset with story-based Question Answering. It is a part of the bAbI project by Facebook Research.

Note: I have currently added all the 160 configs. If this seems impractical, I can keep only a few. While each dummy_data.zip weighs a few KBs, overall it is around 1.3MB for all configurations. This is problematic. Let me know what is to be done.

Thanks :)

Checkbox

• Create the dataset script /datasets/my_dataset/my_dataset.py using the template
• Fill the _DESCRIPTION and _CITATION variables
• Implement _infos(), _split_generators() and _generate_examples()
• Make sure that the BUILDER_CONFIGS class attribute is filled with the different configurations of the dataset and that the BUILDER_CONFIG_CLASS is specified if there is a custom config class.
• Generate the metadata file dataset_infos.json for all configurations
• Generate the dummy data dummy_data.zip files to have the dataset script tested and that they don't weigh too much (<50KB)
• Add the dataset card README.md using the template : fill the tags and the various paragraphs
• Both tests for the real data and the dummy data pass.

[lhoestq]

Impressive one ! good job

[gchhablani]

Hi @lhoestq,

Should I remove the 160 configurations? Is it too much?

EDIT:
Can you also check the task category? I'm not sure if there is an appropriate tag for the same.

[lhoestq]

Thanks for the changes !

Should I remove the 160 configurations? Is it too much?

Yea 160 configuration is a lot.
Maybe this dataset can work with parameters type and task_no ?
You can just remove the configuration in BUILDER_CONFIGS to only keep a few ones.
Also feel free to add an example in the dataset card of how to load the other configurations

load_dataset("babi_qa", type="hn", task_no="qa1")

for example, and with a list of the possible combinations.

Can you also check the task category? I'm not sure if there is an appropriate tag for the same.

It looks appropriate, thanks :)

[gchhablani]

Hi @lhoestq

I'm unable to test it locally using:

load_dataset("datasets/babi_qa", type="hn", task_no="qa1")

It raises an error:

TypeError: __init__() got an unexpected keyword argument 'type'

Will this be possible only after merging? Or am I missing something here?

[lhoestq]

Can you try adding this class attribute to BabiQa ?

BUILDER_CONFIG_CLASS = BabiQaConfig

This should fix the TypeError issue you got

[gchhablani]

My bad. Thanks a lot!

[gchhablani]

Hi @lhoestq

I have added the changes. Only the "qa1" task for each category is included. Also, I haven't removed the size categories and other description because I think it will still be useful. I have updated the line in README showing the example.

Thanks,
Gunjan

[gchhablani]

Hi @lhoestq,

Does this look good now?

[lhoestq]

Thanks !!