Analyzing Dream of the Red Chamber Character Relations with Word Vectors

Preface: For various reasons I keep blending hobbies with work. I often marvel at Cao Xueqin’s ingenuity, language, and moving plot—and wondered: can a machine read the love between “Baoyu” and “Daiyu”?

Data Processing

The data is the great Dream of the Red Chamber itself—a downloaded txt file.

Then convert encoding to UTF-8, segment words, remove lots of spaces and punctuation. Two methods build word vectors: n-gram model training and Word2Vec.

Building Word Vectors with n-gram

Transform data to this format:

That is N=2. Code:

with open("红楼梦.txt", encoding='utf-8') as f:
    text = f.read()
temp = jieba.lcut(text)
words = []
for i in temp:
    i = re.sub("[\s+\.\!\/_,$%^*(+\"\'""《》]+|[+——！，。？、~@#￥%……&*（）：]+", "", i)
    if len(i) > 0:
        words.append(i)
trigrams = [([words[i], words[i + 1]], words[i + 2]) for i in range(len(words) - 2)]
print(trigrams[:3])

Build vocabulary:

vocab = set(words)
word_to_idx = {}
idx_to_word = {}
ids = 0
for w in words:
    cnt = word_to_idx.get(w, [ids, 0])
    if cnt[1] == 0:
        ids += 1
    cnt[1] += 1
    word_to_idx[w] = cnt
    idx_to_word[ids] = w

Model build and training. Structure:

1. Input: embedding layer—map word id to one-hot, linear layer to word vector, output embedding_dim.
2. Linear: embedding_dim → 128, then ReLU.
3. Linear: 128 → vocab size, then log softmax for next-word probabilities.

Full code: n-grama-and-wordvect-Analysis-of-the-text

Training iterations:

Word2Vec Training

Reshape data into sentences, e.g.:

['因此', '大家', '议定', '每日', '轮流', '做', '晚饭', '之主']

['天天', '宰猪', '割羊', '屠鹅', '杀鸭', '好似', '临潼斗宝', '的', '一般', '都', '要', '卖弄', '自己', '家里', '的', '好', '厨役', '好', '烹调']

Processing code:

f = open("红楼梦.txt", encoding='utf-8')
f = f.read().split("。")
lines = []
for line in f:
    temp = jieba.lcut(line)
    words = []
    for i in temp:
        i = re.sub("[\s+\.\!\/_,$%^*(+\"\'""《》]+|[+——！，\- 。？、~@#￥%……&*（）：；'']+", "", i)
        if len(i) > 0:
            words.append(i)
    if len(words) > 0:
        lines.append(words)

Feed into the model and project to 2D for display:

model = Word2Vec(lines, size=20, window=2, min_count=0)
renwu = model.wv.most_similar('林黛玉', topn=20)
rawWordVec = []
word2ind = {}
for i, w in enumerate(model.wv.vocab):
    rawWordVec.append(model[w])
    word2ind[w] = i
rawWordVec = np.array(rawWordVec)
X_reduced = PCA(n_components=2).fit_transform(rawWordVec)

Plot and highlight key characters: Jia Baoyu, Lin Daiyu, Xiangling, Jia Zheng, Qingwen, Miaoyu, Xiren, Xue Baochai, Wang Xifeng, Ping’er, Grandmother Jia, Tanchun.

fig = plt.figure(figsize=(15, 10))
ax = fig.gca()
ax.set_facecolor('black')
ax.plot(X_reduced[:, 0], X_reduced[:, 1], '.', markersize=1, alpha=0.3, color='white')
words = ['贾宝玉', '林黛玉', '香菱', '贾政', '晴雯', '妙玉', '袭人',
         '薛宝钗', '王熙凤', '平儿', '贾母', '探春']
zhfont1 = matplotlib.font_manager.FontProperties(fname='./华文仿宋.ttf', size=16)
for w in words:
    if w in word2ind:
        ind = word2ind[w]
        xy = X_reduced[ind]
        plt.plot(xy[0], xy[1], '.', alpha=1, color='red')
        plt.text(xy[0], xy[1], w, fontproperties=zhfont1, alpha=1, color='yellow')
plt.show()

Result:

Baoyu, Daiyu, Baochai, and Xifeng cluster closely—names almost indistinguishable. Zoom in and—surprise…

Conclusion: the machine also “read” the Baoyu–Daiyu love.

Note: the above uses only Dream of the Red Chamber text. With a huge external corpus (Weibo, People’s Daily, Shanghai Hotline, Autohome, etc.—1,366,130 vectors), results differ. Space is limited—results directly:

All of the above is personal amusement—for shared enjoyment; criticize freely if you dislike it.