Origin/fix missing features error

[eunseojo]

This fixes the problem of when the dataset_load function reads a function with "features" provided but some read batches don't have columns that later show up. For instance, the provided "features" requires columns A,B,C but only columns B,C show. This fixes this by adding the column A with nulls.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[lhoestq]

Awesome thanks !

Could you also add a test in tests/packaged_modules/test_json.py ?

You can copy test_json_generate_tables and rename it something like test_json_generate_tables_with_features, and pass explicit features to Json(...)

[eunseojo]

please review :) @lhoestq @ola13 thankoo

[lhoestq]

Thanks :) I just updated the test to make sure it works even when there's a column missing, and did a minor change to json.py to add the missing columns for the other kinds of JSON files as well (I moved the code toself._cast_table)

[lhoestq]

Thanks for fixing this ! Also cc @mariosasko does it sound good to you as well ?

[ola13]

Thanks Unso! If @lhoestq is happy then I'm also happy :D

[mariosasko]

When I noticed the ping, this PR had already been merged...

Luckily, PyArrow's read_json behaves the same when explicit_schema is given via ParseOptions, so I'm okay with this change (our JSON loader doesn't use read_json for decoding JSON in some scenarios, so this manual approach is the right one).