Machine Learning (15) — Bayesian Networks

Preface:

When multiple features are correlated, naive Bayes cannot handle such cases well; Bayesian networks are a strong approach. Hidden Markov models are the most common application of Bayesian networks.

In general, nodes in a Bayesian network DAG represent random variables—observed, latent, unknown parameters, etc. Arrows between nodes represent causal (non–conditionally independent) relationships. A single arrow from one node to another means one is the “cause” and the other the “effect,” yielding a conditional probability. Note: each node is conditionally independent of its non-descendants given its direct parents.

Simple Bayesian Networks

The key method is the graphical model: connect causally related variables with arrows. DAG nodes are random variables; arrows indicate causal links between variables.

Bayesian networks model uncertain causal reasoning with conditional probabilities and a directed acyclic graph.

The figure below is a simple illustration:

The network above is expressed as:

Fully connected Bayes nets look like:

Compared with ordinary Bayesian networks.

Conditional Independence in Bayesian Networks

1. Given C, a and b are blocked and independent.

Conditional independence: tail-to-tail. As shown:

This can be written as:

2. Given C, a and b are blocked and independent.

Conditional independence: head-to-tail

Same formulas as above.

2. When C is unknown, a and b are blocked and independent.

Conditional independence: head-to-head

Formulas:

Bayesian Summary

Main advantages of naive Bayes: (1) rooted in classical theory with stable classification efficiency; (2) works well on small data, handles multiclass, suits incremental training—especially when data exceeds memory, train in batches; (3) relatively insensitive to missing data, simple, common in text classification.

Main disadvantages: (1) theoretically minimal error rate among classifiers, but not always in practice—naive Bayes assumes feature independence given the class, which often fails with many or correlated features; it works best when correlations are weak. Semi-naive Bayes methods partially address this. (2) requires prior probabilities, often assumption-driven, so poor priors hurt predictions. (3) classification follows from priors and data to posteriors, so errors occur.