Generating Sentence Vectors with BERT

BERT is essentially a two-stage NLP model. Stage one, pre-training, like Word2Vec, trains a language model on unlabeled corpora. Stage two, fine-tuning, uses the pretrained model for specific downstream NLP tasks.

Google has already run large-scale pre-training on expensive hardware. Chinese BERT model: https://storage.googleapis.com/bert_models/2018_11_03/chinese_L-12_H-768_A-12.zip

Two parts: using BERT word/sentence vectors, and fine-tuning for other tasks.

• How to use BERT vectors

• Fine-tuning for text classification

Using BERT Vectors

Traditional sentence vectors average word embeddings and cannot handle polysemy; BERT vectors include context and should theoretically outperform traditional methods.

Method 1: Direct Feature Extraction

1. Download the BERT repo: https://github.com/google-research/bert

extract_features.py generates sentence vectors:

2. Download the Chinese pretrained model: https://storage.googleapis.com/bert_models/2018_11_03/chinese_L-12_H-768_A-12.zip

3. Extract sentence features with parameters:

--input_file="./data/input.txt"
--output_file="./data/output.jsonl"
--vocab_file="./chinese_L-12_H-768_A-12/vocab.txt"
--bert_config_file="./chinese_L-12_H-768_A-12/bert_config.json"
--init_checkpoint="./chinese_L-12_H-768_A-12/bert_model.ckpt"
--layers=-2
--max_seq_length=128
--batch_size=8
layers： 是输出那些层的参数，-1就是最后一层，-2是倒数第二层，一次类推
max_seq_length： 是最大句子长度，根据自己的任务配置。如果你的GPU内存比较小，可以减小这个值，节省存储

Output example:

{"linex_index": 1, "features": [{"token": "[CLS]", "layers": {"index": -1, "values": [-0.2844, 0.450896, 0.285645, 0.421341, 0.411053, ...

Method 2: bert-as-service — Two Lines to Load Vectors

Article: https://zhuanlan.zhihu.com/p/50582974 GitHub: https://github.com/hanxiao/bert-as-service

1. Install bert-as-service:

pip install bert-serving-server  # server
pip install bert-serving-client  # client, independent of `bert-serving-server`

2. Download the Chinese pretrained model (above)

3. Start the service:

bert-serving-start -model_dir D:/数据/实体链接/bert相识度匹配/chinese_L-12_H-768_A-12 -num_worker=1

4. Encode sentences:

from bert_serving.client import BertClient
bc = BertClient()
bc.encode(["今天天气真好","我感冒了"])

Output:

array([[ 0.43153867, -0.22524145,  0.02924719, ..., -0.12929817,
         0.3106631 , -0.1888775 ],
       [ 0.6095807 , -0.2103941 , -0.20782037, ..., -0.24075384,
        -0.25313932, -0.45011818]], dtype=float32)

Method 3: Simple Sentence Vectors Without a Server

Repo: https://github.com/terrifyzhao/bert-utils

from bert.extrac_feature import BertVector
bv = BertVector()
bv.encode(['今天天气不错'])

Output:

[[ 1.21984698e-01  7.84057677e-02 -1.06496774e-01 -3.25891018e-01
   4.94978607e-01 -4.69692767e-01  2.54333645e-01 -8.82656407e-03...