NLP Datasets

A collection of various NLP datasets, mainly Indonesia-related languages. These datasets are split into two: pre-training corpora and fine-tuning datasets.

---

Table of Contents

• Pre-training Corpora
  • Javanese IMDb
  • Javanese Wikipedia v1
  • Javanese Wikipedia v2
  • Sundanese Wikipedia
  • Minangkabau Wikipedia
  • OSCAR
  • CC-100 Monolingual
  • The C4 Multilingual Dataset
• Fine-tuning Datasets
  • Sundanese Twitter Dataset for Emotion Classification
• To Be Added

---

Pre-training Corpora

Javanese IMDb

English IMDb movie review dataset translated to Javanese using multi-lingual MarianMT Transformer Helsinki-NLP/opus-mt-en-mul.

Split (47.5 MB):

• 25,000 Train
• 25,000 Test
• 50,000 Unsupervised

---

Javanese Wikipedia v1

Javanese Wikipedia documents from Wikidump. Collected on December 2020.

Split (319 MB):

• 80,067 Unsupervised

---

Javanese Wikipedia v2

Javanese Wikipedia documents from Wikidump. Collected on June 2021.

Split (342 MB):

• 84,507 Unsupervised

---

Sundanese Wikipedia

Sundanese Wikipedia documents from Wikidump. Collected on June 2021.

Split (279 MB):

• 68,920 Unsupervised

---

Minangkabau Wikipedia

Minangkabau Wikipedia documents from Wikidump. Collected on June 2021.

Split (911 MB):

• 229,869 Unsupervised

---

OSCAR*

"OSCAR or Open Super-large Crawled Aggregated coRpus is a huge multilingual corpus obtained by language classification and filtering of the Common Crawl corpus using the goclassy architecture."

Split^:

• Indonesian (16 GB)
• Javanese (583 KB)
• Sundanese (141 KB)
• Minangkabau (310 KB)

*external resource

^deduplicated

---

CC-100 Monolingual*

"This corpus comprises of monolingual data for 100+ languages [...] This was constructed using the urls and paragraph indices provided by the CC-Net repository by processing January-December 2018 Commoncrawl snapshots."

Split:

• Indonesian (36 GB)
• Javanese (37 MB)
• Sundanese (15 MB)

*external resource

---

The C4 Multilingual Dataset*

"A colossal, cleaned version of Common Crawl's web crawl corpus. Based on Common Crawl dataset."

Split:

• Indonesian (242 GB)
• Javanese (876 MB)
• Sundanese (464 MB)

*external resource

---

Fine-tuning Datasets

Sundanese Twitter Dataset for Emotion Classification*

O. V. Putra, F. M. Wasmanson, T. Harmini, and S. N. Utama, “Sundanese twitter dataset for emotion classification,” virtual, Nov. 2020.

Split (235 KB):

• 2,518 Train
• 12 Test

*external resource

---

TBA

• Wikipedia documents from other regional languages

  • Minangkabau min
  • Banyumasan/Basa Banyumasan map-bms
  • Acehnese ace
  • Gorontalo/Bahasa Hulontalo gor
  • Balinese ban
  • Banjar bjn
  • Madurese mad
  • Toba Batak Language bbc

• Online news

  • Solopos Jagad Jawa (Javanese)
  • BewaraJabar Warta Sunda (Sundanese)

• Classification Datasets

  • Indonesian
    • Tweets Responding to the Indonesian Government’s Handling of COVID-19: Sentiment Analysis Using SVM with Normalized Poly Kernel
  • Javanese
    • Classification of Javanese Language Level on Article Using Multinomial Naive Bayes and N-Gram Methods
    • Attention-based CNN-BiLSTM for dialect identification on Javanese text
  • Sundanese
    • Analisis Sentiment Tweets Berbahasa Sunda Menggunakan Naive Bayes Classifier dengan Seleksi Feature Chi Squared Statistic
    • Sundanese Language Level Detection Using Rule-Based Classification: Case Studies On Twitter