Add long answer candidates to natural questions dataset

[seirasto]

This is a modification of the Natural Questions dataset to include missing information specifically related to long answer candidates. (See here: https://github.com/google-research-datasets/natural-questions#long-answer-candidates). This information is important to ensure consistent comparison with prior work. It does not disturb the rest of the format . @lhoestq @albertvillanova

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[albertvillanova]

Thanks a lot @seirasto for your contribution. Definitely, the addition of this field will be useful for all community members using this dataset.

Just some comments below.

In relation with the non-passing tests, please note that once the changes are validated, we should pre-process all this dataset and upload it to our HuggingFace cloud.

[albertvillanova]

Once we have added long_answer_candidates maybe it would be worth to also add the missing candidate_index (inside long_answer). What do you think, @seirasto ?

[albertvillanova]

Also note the "Data Fields" section in the README is missing the long_answer field.

Moreover, there is no instance example in "Data Instances" section.

[albertvillanova]

We could either make these fixes in this PR or in a subsequent PR.

[seirasto]

@albertvillanova I've added the missing fields and updated the README to include a data instance and some other things.

[albertvillanova]

Thank you.

I think the script is already OK, so that we can start pre-processing the entire dataset: I'm addressing this.

In relation with the dataset card (README file), I think there is an issue: the fields in both "Data Instances" and "Data Fields" section should be aligned (naming and nesting) with the fields we return in the script:

datasets/datasets/natural_questions/natural_questions.py

Lines 187 to 200 in 7129c41

	{
	"id": id_,
	"document": {
	"title": ex_json["document_title"],
	"url": ex_json["document_url"],
	"html": ex_json["document_html"],
	"tokens": [
	{"token": t["token"], "is_html": t["html_token"], "start_byte": t["start_byte"], "end_byte": t["end_byte"]} for t in ex_json["document_tokens"]
	],
	},
	"question": {"text": ex_json["question_text"], "tokens": ex_json["question_tokens"]},
	"long_answer_candidates": [lac_json for lac_json in ex_json["long_answer_candidates"]],
	"annotations": [_parse_annotation(an_json) for an_json in ex_json["annotations"]],
	},

[seirasto]

Great! I've made the updates to align the README. Please let me know if I missed anything.

[albertvillanova]

As there were many minor little fixes, I thought it would be easier to fix them directly.

[albertvillanova]

I think the loading script is OK now. If it is also validated by another datasets maintainer, I could run the generation of the pre-processed data and then merge this PR into master (once all the tests are green).

CC: @lhoestq

[lhoestq]

It looks good to me, thanks @seirasto !

[albertvillanova]

I have merged the master branch, so that we include all the fixes on Apache Beam + Google Dataflow.

[albertvillanova]

Pre-processing is running!

Already finished for "dev" config:

In [2]: ds = load_dataset("datasets/natural_questions", "dev")

In [3]: ds
Out[3]: 
DatasetDict({
    validation: Dataset({
        features: ['id', 'document', 'question', 'long_answer_candidates', 'annotations'],
        num_rows: 7830
    })
})

[albertvillanova]

There is an issue while running the preprocessing for the "default" (train+dev) config. Train data files are larger than than dev ones and workers run out of memory.

I'm opening a separate issue to handle this problem: #4525

[albertvillanova]

@seirasto is proposing uploading their preprocessed data files to our Datasets bucket.

I think @lhoestq can give a more informed answer about authentication requirements.

[lhoestq]

Now that the data fiels are uploaded, can you merge the main branch into yours to re-trigger the CI @seirasto please ? :) Then I think we can merge if it's good for you @albertvillanova

[albertvillanova]

Good, once CI is green! Finally we can merge this PR... 😅

[seirasto]

Merge is done! I think someone needs to approve the CI to run :)

[lhoestq]

Can you run make style to fix the code formatting required by the CI please ?

[albertvillanova]

After running make style, these are the style fixes (see below).

[seirasto]

Thanks @albertvillanova! I've committed all your suggestions.

[albertvillanova]

The CI is green. I'm merging this PR.