ROC Stories
Introduction
ROCStories are simple stories of four sentences. In the test set, two sentences are added to each story, one of which is a meaningful continuation of the story, while the second is not. For example:
Karen was assigned a roommate her first year of college.
Her roommate asked her to go to a nearby city for a concert.
Karen agreed happily. The show was absolutely exhilarating.
- Karen became good friends with her roommate. (right ending)
- Karen hated her roommate. (wrong ending )
Training data contain five sentences (story and its correct continuation).
There is a file
100KStories.csv
(see also the file 100KStories.zip)
with 98'167 stories, totaling 4'859'629 tokens
and 36'566 unique word forms.
For comparison, one volume of "War and Peace" contains about 150'000 tokens,
and wikipedia has several billion.
Loading Stories
Since the dataset is in a csv file, we will use the library pandas. We will save stories in the list docs. Each training story is a list of 5 sentences. The first two columns of the csv file are service columns:
import re # regular expressions
import pandas as pd # csv files
df = pd.read_csv('100KStories.zip', sep=',') # read from zip file
docs = [] # list of stories
for i in range( len(df) ):
sents = [] # list of sentences
for j in range(5):
sents.append( preprocessing( df.iloc[i,2+j]) ) # add sentence
docs.append(sents) # add story
Before saving sentences, a small preprocessing is performed: in the preprocess function, specific spaces, quotes are removed, everything is converted to lowercase and punctuation marks are separated from words:
def preprocess(s):
s = s.translate( {ord(c): ' ' for c in "\u202f\u200b\xa0"} ) # different spaces to ' '
s = s.translate( {ord(c): ' ' for c in "\"«»"} ) # quotes to ' '
s = re.sub( '\s+', ' ', s).strip() # multiple spaces to one
s = s.lower() # to lowercase
res = []
for i, ch in enumerate(s): # separate punctuation
if ch in ".,:;!?…%" and s[i-1] != ' ': res.append(' '+ch)
else: res.append(ch)
return (' '+ "".join(res)+' ' ) # for search like ' cat '
Word Vocabulary
To compile the vocabulary, we will use the Counter object from the standard library collections:from collections import Counter words = [w for d in docs for s in d for w in s.split()] cnt = Counter(words) # vocabulary tokens = len(wrds) # total words in textIn the wordID dictionary we will store the word number (id) and the number of its occurrences per million words (pm), keeping only the most frequent words that appeared at least 12 times:
V_DIM = sum( v >= 12 for v in cnt.values() ) # appeared at least 12 times
wordID = dict( cnt.most_common(V_DIM) ) # take the V_DIM most frequent words
for i,w in enumerate(wordID):
wordID[w] = {"id": i, "pm": int(100000000*(cnt[w]/tokens))/100 }
In ROCStories there are 10'393 such words.
Below are the first 160 words with their frequency pm:
. 97442 with 5202 as 2666 some 1784 bought 1414 put 1117 thought 947 because 826 the 43773 that 4999 an 2562 found 1775 started 1411 family 1107 really 947 game 825 to 34919 up 4792 very 2541 tom 1725 house 1392 no 1089 like 939 buy 820 a 28664 's 4635 them 2435 into 1708 first 1373 just 1086 lot 911 class 814 was 24241 out 4540 not 2289 made 1705 down 1373 always 1079 happy 899 how 810 he 23352 him 4455 home 2273 school 1702 did 1313 looked 1077 has 894 find 806 she 19702 my 4454 from 2216 told 1608 night 1308 money 1044 while 893 great 802 and 19082 one 4322 after 2201 work 1600 finally 1275 said 1039 away 891 ran 797 her 14894 went 4261 we 2178 then 1594 came 1261 make 1036 before 891 john 794 , 14208 day 4183 get 2120 friend 1578 tried 1230 man 1020 gave 880 began 792 his 13346 when 3922 time 2110 car 1567 asked 1227 more 1017 much 874 parents 782 it 12458 but 3892 would 2085 me 1526 off 1201 going 1013 play 866 realized 779 i 10552 got 3884 is 2076 back 1521 never 1186 good 997 what 866 old 778 in 10470 decided 3665 go 1999 could 1514 store 1176 job 989 tim 866 food 771 of 9692 all 3389 took 1969 have 1504 this 1171 next 986 called 863 hard 749 for 8922 were 3307 there 1935 over 1463 mom 1171 needed 976 take 861 mother 744 had 8828 ! 3141 be 1919 their 1463 now 1143 couldn't 964 two 859 again 730 on 7407 so 3035 didn't 1918 saw 1444 dog 1134 every 960 do 855 other 725 they 6570 wanted 3016 friends 1900 by 1439 been 1120 too 957 left 834 last 715 at 5413 new 2776 about 1845 loved 1414 felt 1118 see 950 around 834 wasn't 712List of the last (rare words) with pm=2.46:
sparked, olives, stance, 36, badminton,conclusion, pursuing, reddit, mattered, hitter, carole, blooms, charlene, headband, wander, mushy, otto, loretta, moses, mildly, hummus, woody, freyaWith 4.86 million tokens pm=2.46 means that the word in the corpus appeared 12 = 2.46*4.86 times. So, the word zeus (lily has two white mice , zeus and zeke .) with pm=2.46 appeared in four stories 12 times.
The entropy of the story vocabulary is 6.023.
Compared to larger text corpora, there is a bias towards people's names (stories about some Toms, Lizs, etc.).
Word frequencies, as expected, obey Zipf's law.
For the first 1000 words, frequency decreases as $i^{-1.07},$
where $i$ is the serial number
in order of decreasing frequency. For rarer words, frequency decreases faster.
Below on the graphs in logarithmic scale (on both axes) the dependence $\mathrm{pm}(i)$ is drawn:
