Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add Dataset.from_list #4890

Merged
merged 7 commits into from
Sep 2, 2022
Merged

add Dataset.from_list #4890

merged 7 commits into from
Sep 2, 2022

Conversation

sanderland
Copy link
Contributor

@sanderland sanderland commented Aug 25, 2022

As discussed in #4885

I initially added this bit at the end, thinking filling this field was necessary as it is done in from_dict.
However, it seems the constructor takes care of filling info when it is empty.

if info.features is None:
  info.features = Features(
      {
          col: generate_from_arrow_type(coldata.type)
          for col, coldata in zip(pa_table.column_names, pa_table.columns)
      }
  )

@sanderland sanderland marked this pull request as ready for review August 25, 2022 09:22
@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Aug 25, 2022

The documentation is not available anymore as the PR was closed or merged.

@sanderland
Copy link
Contributor Author

sanderland commented Aug 25, 2022

@albertvillanova it seems tests fail on pyarrow 6, perhaps from_pylist is a v7 method? How do you usually handle these version differences?
Added something that at least works

Copy link
Member

@lhoestq lhoestq left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool ! Thanks a lot :)

I added a few comments:

src/datasets/table.py Outdated Show resolved Hide resolved
src/datasets/arrow_dataset.py Outdated Show resolved Hide resolved
tests/test_dataset_list.py Show resolved Hide resolved
docs/source/loading.mdx Outdated Show resolved Hide resolved
sanderland and others added 3 commits August 30, 2022 12:10
Co-authored-by: Quentin Lhoest <42851186+lhoestq@users.noreply.github.com>
Co-authored-by: Quentin Lhoest <42851186+lhoestq@users.noreply.github.com>
Copy link
Member

@lhoestq lhoestq left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks ! LGTM :)

@lhoestq lhoestq merged commit a50f268 into huggingface:main Sep 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants