Indonesian Dataset

Collections of Indonesian Natural Language Processing dictionaries, corpus, datasets.

Table of contents

• Crawl
  • Kaskus WebText
  • Twitter Puisi
  • Wipedia Links
• Dictionary
  • KBBI
  • Wordlist
• Dump
  • CC-News-ID
  • Javanese Text
  • mC4
  • Twitter
• Multimodal
  • CC3M
  • CC12M
  • YFCC100M OpenAI Subset
• Paraphrase
  • MultiNLI
  • ParaBank
  • ParaNMT
  • PAWS
  • SBERT Paraphrase Data
  • SNLI
• Question Answering
  • SQuAD
  • Mathematics Dataset
• Speech
  • Google TTS
  • Unsupervised
• Summarization
  • Gigaword
  • Reddit TLDR
  • WikiHow
• Translation
  • Europarl
  • EuroPat
  • ParaCrawl

Crawl

Kaskus WebText

Scraped content from Kaskus with 3 cendol or more.

Twitter Puisi

Poems from various user on Twitter.

Wikipedia Links

Scraped content from Wikipedia external links.

Dictionary

KBBI

Scraped Kamus Besar Bahasa Indonesia Daring.

Wordlist

105,226 words from Kamus Besar Bahasa Indonesia.

Dump

CC-News-ID

Processed CC-News (CommonCrawl News). Contains News from 2016-2021 from various news outlets.

Javanese Text

Extracting Javanese text from available data.

mC4

(Hopefully) clean mC4.

Twitter

Twitter dump from ArchiveTeam Twitter Stream Grab. More data will be released soon!

Multimodal

CC3M

Translated Conceptual Captions 3M from Google Research.

CC12M

Translated Conceptual Captions 12M from Google Research, meant to be used for vision-and-language pre-training.

YFCC100M OpenAI Subset

Translated YFCC100M OpenAI Subset used in CLIP dataset ablation.

Paraphrase

MultiNLI

Translated Multi-Genre Natural Language Inference (MultiNLI). The corpus is modeled on the SNLI corpus, but differs in that covers a range of genres of spoken and written text.

ParaBank

Translated ParaBank v2.0 dataset.

ParaNMT

Translated ParaNMT dataset, 5M filtered variant.

PAWS

Translated PAWS dataset from Google Research.

SBERT Paraphrase Data

Various paraphrase datasets compiled by SBERT translated to Indonesian.

SNLI

The Stanford Natural Language Inference (SNLI) corpus, translated to Indonesian.

Question Answering

SQuAD

Translated SQuAD from Stanford University.

Mathematics Dataset

Automatically generated mathematical question and answer pairs. Translated from DeepMind's Mathematics Dataset.

Speech

Google TTS

Automatically generated speech using Google Translate TTS (id-ID-Wavenet-A).

Unsupervised

A large-scale Indonesian speech corpus for pre-training speech recognition model.

Summarization

Gigaword

Translated English Gigaword dataset.

Reddit TLDR

4 Million content-summary pairs translated from Webis-TLDR-17.

WikiHow

Replicate WikiHow: A Large Scale Text Summarization Dataset to Indonesian.

Translation

Europarl

A parallel corpus extracted from the European Parliament web site by Philipp Koehn (University of Edinburgh). The main intended use is to aid statistical machine translation research. (Translated to Indonesian using Google Translate)

EuroPat

Parallel corpora of patents from the United States Patent and Trademark Office and from the European Patent Organisation. (Translated to Indonesian using Google Translate)

ParaCrawl

Massively translated ParaCrawl v.7.1 from OPUS using Google Translate.

License

• All derived dataset from this repository follows the license of the original dataset, unless otherwise noted.
• All original/collected from public internet dataset from this repository are licensed as Creative Commons Attribution-ShareAlike (CC BY-SA). Original data inside the compiled dataset are owned by its respective owner.