Short Text Classification Documentation

Task Goal

Classify clinical trial eligibility criteria. Text comes from real trials; short text from the eligibility module on the Chinese Clinical Trial Registry (http://chictr.org.cn/). Data is public with download links on the site.

Task Description

Short-text classification labels and examples:

Evaluation Metric

Macro-F1 (Average-F1). Ranking by Macro-F1. For n classes C1, …, Ci, …, Cn: Precision Pi = correct predictions as Ci / predictions as Ci. Recall Ri = correct as Ci / true Ci count.

Model Introduction

Experiment Process

Hardware

NVIDIA-SMI 430.26 Driver Version: 430.26 CUDA Version: 10.2 GPU: Tesla P100 * 2 VRAM:36GB CPU:7-core Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz RAM:120GB Disk:2T SSD

Environment

Python 3.8.10 pip install ark-nlp==0.0.2 pip install scikit-learn pip install pandas pip install elasticsearch pip install openpyxl pip install python-Levenshtein

Hyperparameters

argg = {
'model_dir': 'data/model_data',
'model_type': 'bert',
'model_name': 'chinese-bert-wwm-ext',
'task_name': 'ctc',
'output_dir': './data/output/ctc/',
'do_train': True,
'do_predict': False,
'result_output_dir': './data/result',
'max_length': 128,
'train_batch_size': 16,
'eval_batch_size': 16,
'learning_rate': 3e-05,
'weight_decay': 0.01,
'adam_epsilon': 1e-08,
'max_grad_norm': 0.0,
'epochs': 5,
'warmup_proportion': 0.1,
'earlystop_patience': 5,
'logging_steps': 200,
'save_steps': 10,
'seed': 2021,
'device': torch.device("cuda"
if torch.cuda.is_available()
else "cpu")
}

Results

epoch:5

## Results

precision: 0.8520105137135594 - recall: 0.8032168382072119 - f1 score: 0.817622871761937