Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dev/multimodal format & basic processing pipeline for multimodal datasets support #64

Merged
merged 13 commits into from
Nov 13, 2023
Merged

Dev/multimodal format & basic processing pipeline for multimodal datasets support #64

merged 13 commits into from
Nov 13, 2023

Conversation

HYLcool
Copy link
Collaborator

@HYLcool HYLcool commented Nov 2, 2023

  • Support the basic Data-Juicer intermediate format for multimodal datasets. The detailed format information will be clarified later in some docs.
    • Support image loading, new configs, new special tokens, new OP abstraction
  • Add a basic image OP: image_aspect_ratio_filter. -- for basic pipeline test
  • Add two format conversion tools: llava2jd and dj2llava, which support convert LLaVA-like datasets to Data-Juicer format and reverse. -- for basic pipeline test

@HYLcool HYLcool added enhancement New feature or request dj:multimodal issues/PRs about multimodal data processing labels Nov 2, 2023
@HYLcool HYLcool added this to the Multimodal Support milestone Nov 2, 2023
@HYLcool HYLcool self-assigned this Nov 2, 2023
This was linked to issues Nov 2, 2023
[MM] llava2dj & dj2llava tools #58
Closed
[MM] image_aspect_ratio_filter #57
Closed
@HYLcool
Copy link
Collaborator Author

HYLcool commented Nov 2, 2023 •
edited
Loading

This PR is under an internal test now. Don't merge it until it's done.

  • ds2dj -> dj-process -> dj2ds pipeline validation
  • ds2dj -> dj-process -> dj2ds -> training pipeline validation
    • pretrain
    • finetuning

Update 2023/11/06: all pipeline validations are passed; processed and converted datasets are acceptable to the target model.

This was linked to issues Nov 2, 2023
[MM enhancement] image data loading #56
Closed
[MM enhancement] support text-based interleaved multimodal data as the intermediate format #49
Closed
[enhance] refine the log info when input dataset_path is invalid #55
Closed
@HYLcool
* fix typos
1b97081 
+ add new args into config_all.yaml
@HYLcool
+ add args to control whether to convert the absolute paths in the pr…
5fc1f62 
…ocessed dataset to the relative paths like the original dataset
@HYLcool
* avoid unaligned columns when converting and processing the dataset …
473b088 
…due to missing column or unaligned data type
@HYLcool HYLcool mentioned this pull request Nov 8, 2023
Merged
@HYLcool
Merge branch 'main' into dev/mm_fmt
cd817ab 
# Conflicts:
#	data_juicer/format/formatter.py
#	data_juicer/ops/op_fusion.py
@HYLcool
* avoid missing image_key error for text-only datasets
d0e69d7 
+ support to expert to json format (not jsonl!)
@HYLcool HYLcool requested a review from drcege November 10, 2023 07:29
@HYLcool HYLcool requested a review from BeachWang November 10, 2023 07:56
@yxdyc yxdyc linked an issue Nov 13, 2023 that may be closed by this pull request
export_path can not be a folder #66
Closed
2 tasks
zhijianma
zhijianma approved these changes Nov 13, 2023
View reviewed changes
Copy link
Collaborator

@zhijianma zhijianma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

yxdyc
yxdyc reviewed Nov 13, 2023
View reviewed changes
Copy link
Collaborator

@yxdyc yxdyc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

leaved some minor suggestions

configs/config_all.yaml Outdated Show resolved Hide resolved
data_juicer/config/config.py Outdated Show resolved Hide resolved
data_juicer/ops/base_op.py Outdated Show resolved Hide resolved
@HYLcool HYLcool merged commit 16d159f into main Nov 13, 2023
2 checks passed
@HYLcool HYLcool deleted the dev/mm_fmt branch November 15, 2023 03:22
@HYLcool HYLcool added the dj:op issues/PRs about some specific OPs label Jan 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yxdyc yxdyc yxdyc approved these changes

@zhijianma zhijianma zhijianma approved these changes

@chenhesen chenhesen Awaiting requested review from chenhesen

@drcege drcege Awaiting requested review from drcege

@BeachWang BeachWang Awaiting requested review from BeachWang

Assignees

@HYLcool HYLcool

Labels
dj:multimodal issues/PRs about multimodal data processing dj:op issues/PRs about some specific OPs enhancement New feature or request
Projects
None yet
Milestone
Basic Multimodal Support
3 participants
@HYLcool @zhijianma @yxdyc