Writing

Machine Learning (14) — Naive Bayes

Bayesian probability, naive Bayes variants, text classification, and classifier comparison on news data.

Preface: Among classification algorithms, naive Bayes differs from most. Decision trees, KNN, logistic regression, and SVM are discriminative—they learn Y from X directly (decision function or P(Y|X)). Naive Bayes is generative: model the joint P(X,Y), then P(Y|X) = P(X,Y)/P(Y). It is intuitive, fast, and widely used (spam filtering, text classification).

Algorithm idea: probability-based prediction

Logistic regression fits a curve or hyperplane; decision trees split features; SVM maximizes margin. Naive Bayes uses feature probabilities. Bayes idea—informal example: spotting “villains” in movies by appearance, tone, expression. For “smile” among 100 people, “good smiles” vs “bad smiles” differ in probability and frequency—a naive generative story for classification.

Bayes formula

Bayesian vs frequentist: Bayes is old but long secondary to frequentism (Pearson, Fisher). Bayes gained ground where it works well. Core idea: prior + data → posterior. Data is clear; critics attack the prior—often assumed (Normal, Beta, etc.). Bayes works in spam and text classification.

Independence: if X ⊥ Y, P(X,Y) = P(X)P(Y).

Conditional probability:

P(Y|X) = P(X,Y)/P(X), P(X|Y) = P(X,Y)/P(Y)

So P(Y|X) = P(X|Y)P(Y)/P(X).

Concepts:

  • Prior P(A): probability of A without extra evidence (given from history).

  • Likelihood P(B|A): probability of B given A.

  • Posterior P(A|B): revised P(A) after observing B.

  • Law of total probability: if A and A’ partition the space,

image

Bayes with total probability:

image

Example: P(sunny tomorrow) = 70% (prior). P(stars tonight | sunny) = 80%; P(stars | not sunny) = 30%. Stars appear tonight (evidence). Posterior:

P(sunny|stars) = P(sunny)P(stars|sunny)/P(stars) = (0.7×0.8)/(0.7×0.8+0.3×0.3) ≈ 86%.

Rain probability rises from 30% to 14% for tomorrow sunny—updated belief.

Diagram: left rectangle = prior; horizontal line = condition; dark gray share of gray area = posterior.

image

Naive Bayes algorithm

Naive Bayes (NB) assumes feature independence; supervised Bayes theorem. For features x₁…xₘ and class y:

image

Independence:

image

P(x₁,…,xₘ) is constant given x, so:

image

Model:

image

Algorithm

  1. Item x = {a₁,…,aₘ}; classes C = {y₁,…,yₙ}

  2. Compute P(y₁|x), …, P(yₙ|x)

  3. Predict argmax_k P(y_k|x)

Flowchart:

image

Variants by prior/likelihood:

  1. Gaussian Naive Bayes — continuous features, Gaussian P(x|y):

image

Estimate per-class mean and std per feature.

  1. Bernoulli Naive Bayes — binary features:

image

Bernoulli: outcomes 0/1; E(x)=p, Var(x)=p(1−p).

  1. Multinomial Naive Bayes — count/multinomial features, parameters θ_y:

image

Example: text classification

NB suits text (classification, spam). News categorization with sklearn.

API: GaussianNB, MultinomialNB, BernoulliNB. Continuous → Gaussian; multivariate discrete → Multinomial; sparse binary → Bernoulli.

predict, predict_proba, predict_log_proba—max probability class equals predict.

Simple demo:

import numpy as np
X = np.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1], [3, 2]])
Y = np.array([1, 1, 1, 2, 2, 2])
from sklearn.naive_bayes import GaussianNB
clf = GaussianNB()
clf.fit(X, Y)
print("预测的结果是：")
print(clf.predict([[-0.8, -1]]))
print("属于每个类别的概率分别为：")
print(clf.predict_proba([[-0.8, -1]]))
print("属于每个类别的对数转化后的概率分别为：")
print(clf.predict_log_proba([[-0.8, -1]]))
预测的结果是：
[1]
属于每个类别的概率分别为：
[[9.99999949e-01 5.05653254e-08]]
属于每个类别的对数转化后的概率分别为：
[[-5.05653266e-08 -1.67999998e+01]]

[-0.8,-1] predicted class 1 with ~1.0 vs ~5e-8 for class 2.

News classification — compare many classifiers; highlight NB speed.

1. Imports
import numpy as np
from time import time
import matplotlib.pyplot as plt
import matplotlib as mpl

from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.feature_selection import SelectKBest, chi2
from sklearn.linear_model import RidgeClassifier
from sklearn.svm import LinearSVC,SVC
from sklearn.naive_bayes import MultinomialNB, BernoulliNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV
from sklearn import metrics
mpl.rcParams['font.sans-serif'] = [u'SimHei']
mpl.rcParams['axes.unicode_minus'] = False

2. Load 20 newsgroups (4 categories)
print (u'加载数据...')
t_start = time()
remove = ('headers', 'footers', 'quotes')
categories = 'alt.atheism', 'talk.religion.misc', 'comp.graphics', 'sci.space'

data_train = fetch_20newsgroups(data_home='./datas/',subset='train', categories=categories, shuffle=True, random_state=0, remove=remove)
data_test = fetch_20newsgroups(data_home='./datas/',subset='test', categories=categories, shuffle=True, random_state=0, remove=remove)

print (u"完成数据加载过程.耗时:%.3fs" % (time() - t_start))

加载数据...
完成数据加载过程.耗时:2.094s

3. Dataset info
def size_mb(docs):
    return sum(len(s.encode('utf-8')) for s in docs) / 1e6
categories = data_train.target_names
data_train_size_mb = size_mb(data_train.data)
data_test_size_mb = size_mb(data_test.data)
print (u'数据类型：', type(data_train.data))
print("%d文本数量 - %0.3fMB (训练数据集)" % (len(data_train.data), data_train_size_mb))
print("%d文本数量 - %0.3fMB (测试数据集)" % (len(data_test.data), data_test_size_mb))
print (u'训练集和测试集使用的%d个类别的名称：' % len(categories))
print(categories)

数据类型：
2034文本数量 - 2.428MB (训练数据集)
1353文本数量 - 1.800MB (测试数据集)
训练集和测试集使用的4个类别的名称：
['alt.atheism', 'comp.graphics', 'sci.space', 'talk.religion.misc']

4. Split
x_train = data_train.data
y_train = data_train.target
x_test = data_test.data
y_test = data_test.target

Sample article:

image

  1. TF-IDF and stop words
vectorizer = TfidfVectorizer(input='content', stop_words='english', max_df=0.5, sublinear_tf=True)
x_train = vectorizer.fit_transform(data_train.data)
x_test = vectorizer.transform(data_test.data)
print (u'训练集样本个数：%d，特征个数：%d' % x_train.shape)
print (u'停止词:\n')
print(vectorizer.get_stop_words())
feature_names = np.asarray(vectorizer.get_feature_names())

Typical stop words:

image

Features: 2034 samples, 26576 dimensions.

  1. Feature selection (χ², k=1000)
ch2 = SelectKBest(chi2, k=1000)
x_train = ch2.fit_transform(x_train, y_train)
x_test = ch2.transform(x_test)
feature_names = [feature_names[i] for i in ch2.get_support(indices=True)]

Sample features:

image

  1. Benchmark
def benchmark(clf,name):
    print (u'分类器：', clf)
    alpha_can = np.logspace(-2, 1, 10)
    model = GridSearchCV(clf, param_grid={'alpha': alpha_can}, cv=5)
    m = alpha_can.size
    if hasattr(clf, 'alpha'):
        model.set_params(param_grid={'alpha': alpha_can})
        m = alpha_can.size
    if hasattr(clf, 'n_neighbors'):
        neighbors_can = np.arange(1, 15)
        model.set_params(param_grid={'n_neighbors': neighbors_can})
        m = neighbors_can.size
    if hasattr(clf, 'C'):
        C_can = np.logspace(1, 3, 3)
        model.set_params(param_grid={'C':C_can})
        m = C_can.size
    if hasattr(clf, 'C') & hasattr(clf, 'gamma'):
        C_can = np.logspace(1, 3, 3)
        gamma_can = np.logspace(-3, 0, 3)
        model.set_params(param_grid={'C':C_can, 'gamma':gamma_can})
        m = C_can.size * gamma_can.size
    if hasattr(clf, 'max_depth'):
        max_depth_can = np.arange(4, 10)
        model.set_params(param_grid={'max_depth': max_depth_can})
        m = max_depth_can.size
    t_start = time()
    model.fit(x_train, y_train)
    t_end = time()
    t_train = (t_end - t_start) / (5*m)
    print (u'5折交叉验证的训练时间为：%.3f秒/(5*%d)=%.3f秒' % ((t_end - t_start), m, t_train))
    print (u'最优超参数为：', model.best_params_)
    t_start = time()
    y_hat = model.predict(x_test)
    t_end = time()
    t_test = t_end - t_start
    print (u'测试时间：%.3f秒' % t_test)
    train_acc = metrics.accuracy_score(y_train, model.predict(x_train))
    test_acc = metrics.accuracy_score(y_test, y_hat)
    print (u'训练集准确率：%.2f%%' % (100 * train_acc))
    print (u'测试集准确率：%.2f%%' % (100 * test_acc))
    return t_train, t_test, 1-train_acc, 1-test_acc, name

8. Compare classifiers
print (u'分类器的比较：\n')
clfs = [
    [RidgeClassifier(), 'Ridge'],
    [KNeighborsClassifier(), 'KNN'],
    [MultinomialNB(), 'MultinomialNB'],
    [BernoulliNB(), 'BernoulliNB'],
    [RandomForestClassifier(n_estimators=200), 'RandomForest'],
    [SVC(), 'SVM'],
    [LinearSVC(loss='squared_hinge', penalty='l1', dual=False, tol=1e-4), 'LinearSVC-l1'],
    [LinearSVC(loss='squared_hinge', penalty='l2', dual=False, tol=1e-4), 'LinearSVC-l2']
]
result = []
for clf,name in clfs:
    a = benchmark(clf,name)
    result.append(a)
    print ('\n')
result = np.array(result)

Results:

image

  1. Bar chart
result = [[x[i] for x in result] for i in range(5)]
training_time, test_time, training_err, test_err, clf_names = result
training_time = np.array(training_time).astype(np.float)
test_time = np.array(test_time).astype(np.float)
training_err = np.array(training_err).astype(np.float)
test_err = np.array(test_err).astype(np.float)
x = np.arange(len(training_time))
plt.figure(figsize=(10, 7), facecolor='w')
ax = plt.axes()
b0 = ax.bar(x+0.1, training_err, width=0.2, color='#77E0A0')
b1 = ax.bar(x+0.3, test_err, width=0.2, color='#8800FF')
ax2 = ax.twinx()
b2 = ax2.bar(x+0.5, training_time, width=0.2, color='#FFA0A0')
b3 = ax2.bar(x+0.7, test_time, width=0.2, color='#FF8080')
plt.xticks(x+0.5, clf_names)
plt.legend([b0[0], b1[0], b2[0], b3[0]], (u'训练集错误率', u'测试集错误率', u'训练时间', u'测试时间'), loc='upper left', shadow=True)
plt.title(u'新闻组文本数据分类及不同分类器效果比较', fontsize=18)
plt.xlabel(u'分类器名称')
plt.grid(True)
plt.tight_layout(2)
plt.show()

image