Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Python][Dataset] Update Python dataset api for LakeSoul #325

Merged
merged 42 commits into from
Sep 14, 2023
Merged

[Python][Dataset] Update Python dataset api for LakeSoul #325

merged 42 commits into from
Sep 14, 2023

Conversation

codingfun2022
Copy link
Contributor

Part of #201

  1. Support configuring batch_size (default to 16) and thread_num (default to 1; use 0 for hardware concurrency).
  2. Support configuring partition info and namespace.
  3. Add HuggingFace Dataset api.
  4. Support multi-node PyTorch training job.
  5. Add script to build cibuildwheel docker image and build Python wheels with cibuildwheel.

Xiong HuaiDong and others added 30 commits August 24, 2023 15:48
Add LakeSoul arrow dataset Python interface.
ad98cc3 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Call cargo to build liblakesoul_metadata_c.so.
5f1a7bc 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Build and install shared libraries into python directory.
352aecc 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Move metadata into lakesoul directory.
08267f5 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Use liblakesoul_metadata_c.so in .libs dir.
2e38cd5 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Use relative import for lakesoul.metadata.
d18b919 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Move poc.py out of lakesoul.metadata.
c5277a0 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Rename poc.py -> metadata_poc.py.
fcb03fd 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Fix metadata_poc.py imports.
fd9f056 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Remove requirement.txt; update build-requirements.txt.
911b1ee 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Generate .py from .proto and install .py files.
6d26e8b 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Comment out prints in callbacks.
1c4a117 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Remove compile_pb.sh.
da4dc81 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Add pyproject.toml for Python wheel.
b5af4b6 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Fix metadata Python imports.
f7bac8f 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Remove local data test code.
fb24074 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Fix MANIFEST.in for shared libraries.
c1410d2 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Add script build_wheel.py.
87955f9 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Add get_pg_conf_from_env.
69166e3 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Add lakesoul wheel optional dependencies.
12c7bc2 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Merge lakesoul-spark.
056ed20 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Merge branch 'main' into dev/lakesoul_dataset
80c06a1
Fix NativeMetadataClient.__del__.
8d4a340 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Function lakesoul_dataset supports partitions and namespace.
649a486 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
add arg exclude_partition at get_arrow_schema_by_table_name
686798f 
Signed-off-by: zenghua <huazeng@dmetasoul.com>
Wrap LakeSoulDataset to make the dataset object serializable.
a82b556 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Merge remote-tracking branch 'Ceng23333/dev/python_reader' into dev/l…
8bf002b 
…akesoul_dataset
Fix retain_partition_columns.
d867db5 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Support configuring batch_size and thread_count.
f89fd4e 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Raise exceptions for nonexistent LakeSoul tables.
2581097 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Xiong HuaiDong added 12 commits August 31, 2023 16:27
Forward the schema property.
e5389e7 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Implement HuggingFace datasets.IterableDataset.from_lakesoul method.
e81888c 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Support multiple PyTorch worker nodes.
513b992 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Fix LakeSoul python wheel version.
58db5b4 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Add LakeSoul python wheel build env Dockerfile.
8e0fd84 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Update to use devtoolset-11.
70cbeed 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Update setup.py to provide default .so and dir paths.
e16371a 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Fix cibuildwheel repair-wheel-command.
253b0d6 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Rename to host_build_wheel.py.
bc64a96 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Add script docker_build_all_wheels.py.
75c5b94 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
Merge branch 'main' into dev/lakesoul_dataset
6db9a11 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>

Conflicts:
	python/lakesoul/metadata/db_manager.py
	python/lakesoul/metadata/utils.py
Fix Rust subproject paths and build errors.
4ebe70f 
Signed-off-by: Xiong HuaiDong <xionghd@dmetasoul.com>
@xuchen-plus xuchen-plus merged commit ff85b49 into lakesoul-io:main Sep 14, 2023
@codingfun2022 codingfun2022 deleted the dev/lakesoul_dataset branch September 14, 2023 11:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xuchen-plus xuchen-plus xuchen-plus approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@codingfun2022 @xuchen-plus