Writing

NLP (1) — Word Vectors

RNNs handle ordered sequences—we live in an ordered world: time, music, sentences, even a match like the World Cup.

Preface: Deep RNNs handle ordered problems; we live in an ordered world—time, music, spoken sentences, even a game like the recent World Cup in Russia.

One-Hot Encoding

In classification we often use one-hot encoding. Treating each word in natural language as a class yields huge dimensionality but solves the basics—distinguishing words. As shown:

image

That produces 14,901 dimensions. Problems: too much space; no similarity between words—sparsity. One-hot code:

from sklearn.preprocessing import OneHotEncoder
# lables = ['ni','号','ni','meimei']
lables = [0,1,0,4]
lables = np.array(lables).reshape(len(lables),-1)
enc =   OneHotEncoder()
enc.fit(lables)
target = enc.transform(lables).toarray()

print(target)

Output:

image

Word Vector Idea

We need encodings like this:

image

That addresses the issues above. Many methods follow; two main hypotheses:

  • Hypothesis 1: The distributional hypothesis

A word is inferred from surrounding words; similar words appear in similar contexts. Example: “There are many stars in the sky tonight.” Sky and stars co-occur horizontally; context indicates similarity. BOW, LSI, LDA, PMI/PPMI, SVD, etc. build on this—1954 BOW (no order, count +1); 2003 LDA (topic model special case).

  • Hypothesis 2: Distributed models

Similar words in similar contexts. “There are many stars in the sky today” vs “There is a sun in the sky today”—stars and sun share context but are vertically similar. Word2Vec is the main method here. Word2Vec is actually a byproduct of neural language modeling.

Word2Vec

Word2vec is a project name—a tool for word embeddings, combining CBOW and Skip-Gram, fully open source. CBOW predicts the current word from context; Skip-Gram predicts context from the current word.

image

Embedding

Input to hidden layer in Word2Vec is embedding—dimensionality reduction from high-dimensional one-hot, trainable, fully connected, configurable length, fewer parameters.

image

Skip-Gram Principle

image

Skip-Gram adds an output layer on embedding: given a word, compute probabilities of nearby words. For “你真漂亮” we get four one-hot inputs: 1000, 0100, 0010, 0001—possible CBOW inputs; suppose input is 0100 (“真”).

Outputs: two vectors if we take one word before and after. Each output is a probability vector, e.g. (0.3, 0.5, 0.7) and (0.1, 0.9, 0.1)—probabilities of “你”, “真”, “漂亮” before/after “真”. Monte Carlo sampling from these gives a concrete context. Optimize so probability of “真漂亮” in context is high. Objective:

image

T is vocabulary size; p(wt+j|wt) is log-probability of context given wt.

image

Negative Sampling

Full softmax is too expensive; negative sampling helps. Random word combinations are unlikely in real text, so the model should assign low probability to negatives and high probability to true context. New objective:

image