NLP: Datasets

Overview of NLP Tasks

The ultimate goal of NLP is free communication with an intelligent system in natural language (dialogue, obtaining information, setting tasks, etc.). For testing various models, formal specific tasks are chosen, which can be divided into three large groups: classification tasks, labeling tasks, and generation tasks.

Classification Tasks

Labeling Tasks

Generation Tasks

1 Billion Word Language Model Benchmark

The English-language text corpus is described in the article Chelba C., et.al (2014). Its vocabulary consists of 793471 words and has a length of 0.8 billion tokens. Words outside the vocabulary are marked (?) as <UNK> (there are 0.28% of them). The texts are divided into sentences, in which tokenization has been performed. Sentences are shuffled: 

The journalist 's job is to help inform the public on issues that matter .
The golfer , still the highest-paid athlete in the world , was second last year .
Anderson scored 12 points and Amanda Nisleit added 11 points and 12 rebounds .

For downloading, the corpus is available at this link or at this and occupies 1.7 Gb in the archive. It is divided into files.

ROC Story Cloze

ROCStories are coherent simple stories of four sentences. In the test set, two sentences are added to each story, one of which is a meaningful continuation of the story, while the second is not. For example:

Karen was assigned a roommate her first year of college.
Her roommate asked her to go to a nearby city for a concert.
Karen agreed happily. The show was absolutely exhilarating.

There is a file 100KStories.csv with 98'167 stories, totaling 4'859'629 tokens and 36'566 unique word forms.

MultiNLI

MultiNLI - Natural Language Inference corpus with 433k sentence pairs

Stanford Natural Language Inference (SNLI)

SNLI Corpus consists of 570k pairs of English sentences in which the second sentence either follows from the first (entailment), or contradicts (contradiction) or is neutral (neutral). 
Text      : A man inspects the uniform of a figure in some East Asian country.   
Judgments : contradiction  CCCCC
Hypothesis: The man is sleeping

Text      : A soccer game with multiple males playing.   
Judgments : entailment    EEEEE
Hypothesis: Some men are playing a sport.

Text      : An older and younger man smiling.   
Judgments : neutral NNENN 
Hypothesis: Two men are smiling and laughing at the cats playing on the floor.

Stanford Question Answering Dataset (SQuAD)

SQuAD - a dataset for reading comprehension, consisting of questions posed by people on a set of Wikipedia articles, where the answer to each question is a fragment of text or a span from the corresponding reading passage. The question may also be unanswerable. 
"data": [
   {"title": "Beyonc\u00e9", 
    "paragraphs": [{"qas": [       
       {"question": "When did Beyonce start becoming popular?",         
        "answers": [{"text": "in the late 1990s", "answer_start": 269}], 
        "is_impossible": false}, 
    
       {"question": "What areas did Beyonce compete in when she was growing up?",     
        "answers": [{"text": "singing and dancing", "answer_start": 207}], 
        "is_impossible": false}, 
        
       {"question": "In what city and state did Beyonce  grow up? ",     
        "answers": [{"text": "Houston, Texas", "answer_start": 166}], 
        "is_impossible": false},

Stanford Sentiment Analysis

Sentiment Treebank for predicting the sentiment of movie reviews. Data can be downloaded here. 
hide new secretions from the parental units  0
contains no wit , only labored gags    0
that loves its characters and communicates something rather beautiful about human nature  1

General Language Understanding Evaluation (GLUE)

This benchmark is a collection of eleven language understanding tasks: The Corpus of Linguistic Acceptability, The Stanford Sentiment Treebank, Microsoft Research Paraphrase Corpus, Semantic Textual Similarity Benchmark, Quora Question Pairs, MultiNLI Matched, MultiNLI Mismatched, Question NLI, Recognizing Textual Entailment, Winograd NLI, Diagnostics Main.

The download page contains text data separately for each task.

CommonsenseQA

CommonsenseQA is a new multiple-choice question answering dataset that requires different types of commonsense knowledge to predict the correct answers . It contains 12,102 questions with one correct answer and four distractor answers. The dataset is provided in two major training/validation/testing set splits: "Random split" which is the main evaluation split, and "Question token split", see paper for details.

Literature