ML: Pre-trained Embedding Vectors

---

GloVe for English

For English, you can download ready-made word vectorizations with various dimensions E_DIM = E, for example GloVe. The GloVe file is a regular text file, and loading it looks like this:

vec = {}
with open('glove.6B.100d.txt', "r", encoding='utf-8') as file:
    for line in file:
        values = line.split()
        vec[ values[0] ] = np.asarray(values[1:], dtype='float32')

This file contains 400000 words, sorted by frequency. The first 100 are shown below (separated by a line):

   |the|,|.|of|to|and|in|a|"|'s|for|-|that|on|is|was|said|with|he|as|it|by|at|(|)|from|his|''|
   |``|an|be|has|are|have|but|were|not|this|who|they|had|i|which|will|their|:|or|its|one|after|
   |new|been|also|we|would|two|more|'|first|about|up|when|year|there|all|--|out|she|other|
   |people|n't|her|percent|than|over|into|last|some|government|time|$|you|years|if|no|world|
   |can|three|do|;|president|only|state|million|could|us|most|_|against|u.s.|

The vectors of more frequent words have a greater length than rare words (histograms for the first and last 5000 words):

If we consider frequent and rare words separately, then the distance np.linalg.norm(v1-v2) between vectors decreases as the cosine of the angle between them increases (below for 1000 words and a combined image for frequent and rare words):

In this sense, there is no particular difference between these two metrics. Below are the nearest neighbors to a given word according to the cosine metric $1-\cos(\mathbf{v}_1,\mathbf{v}_2)$, with the value shown in parentheses:

   man:                boy               book              ball               pen             
   woman  (.17)        girl   (.08)      books    (.15)    kick   (.23)       pencil   (.39)
   boy    (.21)        man    (.21)      novel    (.18)    throw  (.26)       pens     (.40)
   one    (.22)        kid    (.22)      published(.20)    balls  (.27)       ballpoint(.40)
   person (.25)        woman  (.23)      story    (.21)    off    (.27)       pens     (.40)             
   another(.25)        boys   (.24)      author   (.21)    pitch  (.28)                     
   old    (.26)        child  (.25)      wrote    (.21)    bounced(.30)                     

   france              paris             monday             january           physics  
   belgium    (.19)    france    (.25)   tuesday  (.01)    december (.01)     chemistry   (.15)
   french     (.20)    london    (.27)   thursday (.01)    october  (.01)     mathematics (.17)
   britain    (.20)    brussels  (.30)   wednesday(.01)    february (.01)     science     (.21)
   spain      (.24)    french    (.31)   friday   (.01)    november (.01)     biology     (.21)
   paris      (.25)    rome      (.31)   saturday (.08)    september(.02)     theoretical (.27)
   germany    (.27)    amsterdam (.32)   sunday   (.08)    june     (.02)     astronomy   (.28)
   italy      (.28)    vienna    (.34)   week     (.11)    august   (.02)     sciences    (.28)
   europe     (.29)    berlin    (.34)   earlier  (.15)    july     (.02)     biochemistry(.29)
   netherlands(.29)    madrid    (.37)   afternoon(.16)    april    (.03)     mathematical(.29)
   luxembourg (.32)    strasbourg(.37)   month    (.16)    march    (.04)     astrophysics(.31)

   red                 small             hot               sweet              good 
   yellow(.14)         large  (.07)      cool (.23)        delicious(.30)     better(.11)
   blue  (.16)         larger (.17)      cold (.27)        flavor   (.32)     sure  (.17)
   green (.18)         smaller(.17)      warm (.30)        salty    (.34)     really(.17)
   black (.23)         tiny   (.18)      heat (.32)        taste    (.34)     kind  (.17)
   white (.23)         few    (.23)      soft (.34)        honey    (.35)     very  (.17)
   purple(.23)         well   (.26)      dry  (.34)        spicy    (.35)     we    (.18)

   go                  run               give              say                think   
   going (.06)         running(.20)      take   (.12)      believe(.10)       know  (.08) 
   come  (.07)         runs   (.23)      make   (.13)      why    (.13)       really(.09) 
   get   (.12)         ran    (.25)      giving (.14)      n't    (.13)       why   (.09) 
   'll   (.12)         out    (.26)      put    (.16)      want   (.14)       n't   (.09) 
   do    (.13)         go     (.27)      get    (.18)      know   (.14)       we    (.10) 
   take  (.13)         third  (.27)      enough (.18)      what   (.14)       sure  (.11) 
   you   (.14)         allowed(.28)      want   (.18)      not    (.15)       're   (.11)
   n't   (.14)         first  (.28)      to     (.19)      might  (.15)       maybe (.12)
   want  (.14)         second (.28)      need   (.19)      did    (.15)       i     (.12)
   let   (.14)         start  (.28)      gives  (.19)      think  (.16)       you   (.12)

Interesting neighbors:

   apple       ~  microsoft(.26), ibm (.32),  intel(.32),      software(.32),    dell (.33)
   blackberry  ~  iphone(.28),    ipad(.30),  smartphone(.32), smartphones(.35)

The vector space contains certain semantic directions, leading to the following vector arithmetic:

   man   - woman  + girl  ~ boy    (.11), man (.16), girl  (0.17), kid    (.22), boys  (.28)
   woman - man    + boy   ~ girl   (.09), boy (.14), woman (0.16), mother (.23), child (.25)
   uk    - london + paris ~ france (.28), uk  (.33)

---

fastText

---