Writing

NLP (2) — Chinese Word Segmentation

Segmentation concepts, dictionary and statistical methods, CRF, HMM, and a Genius example.

The previous article covered word vectors; how do we segment text so computers understand sentences better?

What Is Segmentation

Splitting text into words. Hard parts: 1) ambiguity—e.g. “白开水不如果汁甜”—avoid grouping “如果”. 2) unknown words and POS (person, place). Disambiguation via n-gram or probability—HMM and CRF below.

Method Categories

  • Dictionary-based (mechanical): match against a large lexicon with scan direction (forward/backward) and match rule (max/min). Forward maximum matching: if max word length is Maxlen, take left substring of length Maxlen, match; on failure drop last char and retry until done.

  • Statistical and understanding-based: stable character combinations—co-occurrence frequency suggests words. Models: mutual information, n-gram, neural nets, HMM.

Below: CRF and HMM for Chinese segmentation.

CRF

  • Principle “Sequence labeling.” Tags: Begin (B), Middle (M), End (E), Single (S). CRF segments by joining B–E and S. Learning: feature configs (current word, previous word, etc.) output 1/0; global optimization. Inference: score labels, weighted sum, highest wins. Training: linear-chain CRF like linear-chain HMM—Viterbi dynamic programming. Optimize ∑ weighted features per position.

  • Viterbi Compute first-state tag probs; for each next state keep best previous tag and score; backtrack from end.

  • vs HMM 1) HMM assumes Markov independence; CRF does not—more context. 2) CRF is globally optimal. 3) CRF models joint P(Y|X); HMM models next state given current. 4) CRF depends heavily on features; heavier training.

  • Example Genius (CRF-based Python segmenter):

#encoding=utf-8
import genius
text = u"""昨天,我和施瓦布先生一起与部分企业家进行了交流,大家对中国经济当前、未来发展的态势、走势都十分关心。"""
seg_list = genius.seg_text(
    text,
    use_combine=True,
    use_pinyin_segment=True,
    use_tagging=True,
    use_break=True
)
print('\n'.join(['%s\t%s' % (word.text, word.tagging) for word in seg_list]))
['昨天', ',', '我', '和', '施瓦布', '先生', '一起', '与', '部分', '企业家', '进行', '了', '交流', ',', '大家', '对', '中国', '经济', '当前', '、', '未来', '发展', '的', '态势', '、', '走势', '都', '十分关心']

HMM Segmentation

HMM: temporal probability model—hidden Markov chain generates unobserved states, states emit observations. Dual random process. Elements: hidden states S, observations O, π, transition A, emission B (confusion matrix); π and A define state sequence, B observations—HMM triple:

image

Theory in a dedicated chapter. Code: https://github.com/tostq/Easy_HMM