Code

Fix redpajama links#201

Code

Merged

chenhesen merged 37 commits intomainmodelscope/data-juicer:mainfrom fix_redpajama_linkmodelscope/data-juicer:fix_redpajama_linkCopy head branch name to clipboardJan 30, 2024

+27-27

Commits

Commits on Nov 16, 2023

• fix opencc serialization error

  chenhesen

  committed
  573a704

  Copy full SHA for 573a704

Commits on Nov 20, 2023

• Merge branch 'main' of github.com:alibaba/data-juicer

  chenhesen

  committed
  988224f

  Copy full SHA for 988224f

Commits on Nov 21, 2023

• Merge branch 'main' of github.com:alibaba/data-juicer

  chenhesen

  committed
  0ac51cc

  Copy full SHA for 0ac51cc

• support audio-text data reading

  chenhesen

  committed
  4fee9a1

  Copy full SHA for 4fee9a1

Commits on Nov 22, 2023

• update multimodal_README

  chenhesen

  committed
  d856a80

  Copy full SHA for d856a80

Commits on Nov 23, 2023

• Merge branch 'main' of github.com:alibaba/data-juicer

  chenhesen

  committed
  29de3de

  Copy full SHA for 29de3de

• fix pre-commit error

  chenhesen

  committed
  fc98733

  Copy full SHA for fc98733

• modify audio_special_token

  chenhesen

  committed
  539f099

  Copy full SHA for 539f099

• Merge branch 'main' of github.com:alibaba/data-juicer

  chenhesen

  committed
  ca34dfc

  Copy full SHA for ca34dfc

• support only one target_field

  chenhesen

  committed
  4c27643

  Copy full SHA for 4c27643

• fix pre-commit

  chenhesen

  committed
  e54d197

  Copy full SHA for e54d197

Commits on Nov 24, 2023

• add id for log

  chenhesen

  committed
  4743e62

  Copy full SHA for 4743e62

Commits on Nov 29, 2023

• fix conflict

  chenhesen

  committed
  6c58bee

  Copy full SHA for 6c58bee

Commits on Dec 4, 2023

• Merge branch 'main' of github.com:alibaba/data-juicer

  chenhesen

  committed
  6449f15

  Copy full SHA for 6449f15

Commits on Dec 8, 2023

• Merge branch 'main' of github.com:alibaba/data-juicer

  chenhesen

  committed
  70b6c73

  Copy full SHA for 70b6c73

Commits on Dec 15, 2023

• Merge branch 'main' of github.com:alibaba/data-juicer

  chenhesen

  committed
  4105d1e

  Copy full SHA for 4105d1e

Commits on Dec 21, 2023

• Merge branch 'main' of github.com:alibaba/data-juicer

  chenhesen

  committed
  027af2b

  Copy full SHA for 027af2b

• add remove_repeat_sentences_mapper

  chenhesen

  committed
  8a4a0e7

  Copy full SHA for 8a4a0e7

Commits on Dec 22, 2023

• Merge branch 'main' of github.com:alibaba/data-juicer

  chenhesen

  committed
  65bee64

  Copy full SHA for 65bee64

Commits on Dec 27, 2023

• Merge branch 'main' of github.com:alibaba/data-juicer

  chenhesen

  committed
  05627ed

  Copy full SHA for 05627ed

• modify mapper op number

  chenhesen

  committed
  021d14b

  Copy full SHA for 021d14b

• Merge branch 'main' of github.com:alibaba/data-juicer

  chenhesen

  committed
  584783e

  Copy full SHA for 584783e

Commits on Jan 5, 2024

• update image_blur

  chenhesen

  committed
  1688099

  Copy full SHA for 1688099

• Merge branch 'main' of github.com:alibaba/data-juicer

  chenhesen

  committed
  5b72309

  Copy full SHA for 5b72309

Commits on Jan 11, 2024

• add image_blur_mapper

  chenhesen

  committed
  fb3188a

  Copy full SHA for fb3188a

Commits on Jan 17, 2024

• add image_blur_mapper

  chenhesen

  committed
  8b3d87d

  Copy full SHA for 8b3d87d

• precommit

  chenhesen

  committed
  d80d868

  Copy full SHA for d80d868

• update __init__

  chenhesen

  committed
  3e264b6

  Copy full SHA for 3e264b6

• Merge branch 'main' of github.com:alibaba/data-juicer

  chenhesen

  committed
  1bf1c7d

  Copy full SHA for 1bf1c7d

• fix conflicts

  chenhesen

  committed
  58ae2c6

  Copy full SHA for 58ae2c6

Commits on Jan 18, 2024

• fix conficts

  chenhesen

  committed
  22021b7

  Copy full SHA for 22021b7

Commits on Jan 19, 2024

• replaced by the latest load_data_with_context

  chenhesen

  committed
  344e240

  Copy full SHA for 344e240

• fix docs conflicts

  chenhesen

  committed
  a52311f

  Copy full SHA for a52311f

• fix Operators_ZH

  chenhesen

  committed
  5a6b552

  Copy full SHA for 5a6b552

• fix docs conflicts

  chenhesen

  committed
  4794a74

  Copy full SHA for 4794a74

Commits on Jan 22, 2024

• Merge branch 'main' of github.com:alibaba/data-juicer

  chenhesen

  committed
  124c656

  Copy full SHA for 124c656

Commits on Jan 30, 2024

• fix redpajama link

  chenhesen

  committed
  402f4bf

  Copy full SHA for 402f4bf