Include entity positions as feature in ReCoRD

[richarddwang]

https://huggingface.co/datasets/super_glue/viewer/record/validation

TLDR: We need to record entity positions, which are included in the source data but excluded by the loading script, to enable efficient and effective training for ReCoRD.

Currently, the loading script ignores the entity positions ("entity_start", "entity_end") and only records entity text. This might be because the training method of the official baseline is to make n training instance from a datapoint by replacing "@ placeholder" in query with each entity individually.

But it increases the already heavy computation by multiple folds. So DeBERTa uses a method that take entity embeddings by their positions in the passage, and thus makes one training instance from one data point. It is way more efficient and proved effective for the ReCoRD task.

Can anybody help me with the dataset card rendering error? Maybe @lhoestq ?

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[lhoestq]

Thanks @richarddwang ! Sorry for getting back to you after such a long delay.

Anyway it looks good to me ! Can you just remove the remaining jsonl files that are outside of the dummy data zip files ? They are not needed.

Also can you update the dataset_infos.json file with the new column ?

datasets-cli test ./datasets/super_glue --name record --save_infos

[richarddwang]

Thanks for the reply @lhoestq !

I have sucessed on datasets-cli test ./datasets/super_glue --name record --save_infos,
But as you can see, the check ran into FAILED tests/test_dataset_cards.py::test_changed_dataset_card[super_glue] - V....
How can we solve it?

[lhoestq]

Thanks !

Actually I just noticed that this is a breaking change: the length of the "entities" field changes (current implementation uses a set to dedupe them). What about keeping "entities" deduplicated, and having a "entities_spans" field consisting of a list of {"text": ..., "start": ..., "end": ...} ? This way it won't break users code, e.g.

• https://github.com/google-research/text-to-text-transfer-transformer/blob/3c58859b8fe72c2dbca6a43bc775aa510ba7e706/t5/data/preprocessors.py#L925
• equal operation to perform unbatch for huggingface datasets  #2767 (comment)

What do you think ?

BTW the CI failure is unrelated to this PR - it appears that some tags are missing in the dataset cards ('annotations_creators', 'language_creators', 'license', 'multilinguality', 'size_categories', 'source_datasets', 'task_categories', and 'task_ids'), and so the dataset is not indexed properly on the Hugging Face website. This can be fixed in another PR

[richarddwang]

That would be neat! Let me implement it.

[lhoestq]

Thank you ! LGTM

Merging since the CI errors are unrelated to this PR