Skip to content

Code used for the creation of OBELICS, an open, massive and curated collection of interleaved image-text web documents, containing 141M documents, 115B text tokens and 353M images.

License

Notifications You must be signed in to change notification settings

huggingface/OBELICS

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

20 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

OBELICS

OBELICS is an open, massive and curated collection of interleaved image-text web documents, containing 141M documents, 115B text tokens and 353M images.

Dataset page: https://huggingface.co/datasets/HuggingFaceM4/OBELICS

Visualization of OBELICS web documents: https://huggingface.co/spaces/HuggingFaceM4/obelics_visualization

Paper: https://arxiv.org/abs/2306.16527

Goal and organization of obelics

The folder obelics is aimed for:

The primary techniques are defined in the sub-folder processors, while their invocation is found in callers. The configs used for the extraction and the filtering of the documents are in configs.

We refer to our paper for details about these steps.

In visualization, there are different streamlit visualizations:

Goal and organization of build_obelics

In the folder build_obelics, we are giving all the scripts that were used for the creation of OBELICS, with numbers indicating the chronology.

These scripts often call methods defined in processors but not only, and also define other useful methods.

Citation

If you are using this dataset or this code, please cite

@misc{laurencon2023obelics,
      title={OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents},
      author={Hugo Laurençon and Lucile Saulnier and Léo Tronchon and Stas Bekman and Amanpreet Singh and Anton Lozhkov and Thomas Wang and Siddharth Karamcheti and Alexander M. Rush and Douwe Kiela and Matthieu Cord and Victor Sanh},
      year={2023},
      eprint={2306.16527},
      archivePrefix={arXiv},
      primaryClass={cs.IR}
}

About

Code used for the creation of OBELICS, an open, massive and curated collection of interleaved image-text web documents, containing 141M documents, 115B text tokens and 353M images.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published