Sentence Similarity

Approach 1: Sentence Vectors, Then Cosine Similarity

1. Sentence Vectors for Two Sentences

• Word-frequency vectors Using term frequency ignores synonyms, semantics, and scales poorly on long texts (e.g. 10k-word articles)—huge sparse dimensions, costly and inefficient.

• TF-IDF sentence vectors Reduces dimensionality but still misses semantics.

• Deep learning captures semantics; see Generating Sentence Vectors with BERT.

• Traditional sentence vectors

Sum word vectors and average:

def sent2vec(s):
    words = s
    M = []
    for w in words:
        try:
            M.append(w2v.wv[w])
        except:
            continue
    M = np.array(M)
    v = M.sum(axis=0)
    return v / np.sqrt((v ** 2).sum())

2. Cosine Similarity Between Vectors

####计算余弦夹角
def cos_sim(vector_a, vector_b):
    """
    计算两个向量之间的余弦相似度
    :param vector_a: 向量 a
    :param vector_b: 向量 b
    :return: sim
    """
    vector_a = np.mat(vector_a)
    vector_b = np.mat(vector_b)
    num = float(vector_a * vector_b.T)
    denom = np.linalg.norm(vector_a) * np.linalg.norm(vector_b)
    cos = num / denom
    sim = 0.5 + 0.5 * cos
    return sim

Approach 2: Word Vectors and Word Mover’s Distance (WMD)

Word Mover’s Distance

Word2Vec maps words to vectors where similar words are close. Word Mover’s Distance (WMD) uses that: Euclidean distance between any pair of word vectors across two documents, then weighted sum.

Weight matrix T is somewhat like an HMM transition matrix, but probabilities become weights.

Suppose “Obama” has weight 0.5 in document 1 (from frequency or TF-IDF). Because “Obama” and “president” are similar, much weight (e.g. 0.4) flows from “Obama” to “president”; other words in doc 2 get less. Constraint: outflow from word i in doc 1 equals i’s weight in doc 1; inflow to word j in doc 2 equals j’s weight in doc 2.

References:

Supervised Word Mover’s Distance — NIPS 2016 paper selection #2

https://blog.csdn.net/qrlhl/article/details/78512598

https://blog.csdn.net/weixin_40547993/article/details/89475630

• WMD code:

from gensim.models import KeyedVectors
import jieba
import time
import os

start = time.time()
model = KeyedVectors.load_word2vec_format('vectors.bin', binary=True, unicode_errors='ignore')
end = time.time()
print('Cell took %.2f seconds to run.' % (end - start))
s01= "今天是个好日子"
s02= "今天是晴天"
s03= "天气多云转阴"
sentence_01 = list(jieba.cut(s01))
sentence_02= list(jieba.cut(s02))
sentence_03= list(jieba.cut(s03))

Results:

Cell took 16.98 seconds to run.
Loading model cost 0.701 seconds.
distance = 24.3276
Prefix dict has been built succesfully.
distance = 55.7726