Writing

How to Understand Inductive Bias

What inductive bias is and common examples in machine learning and deep learning.

What Is Inductive Bias

When training machine learning tasks, we make assumptions—for example, in KNN we assume nearby points in feature space tend to belong to the same class; in SVM we assume a good classifier should maximize the margin between classes; and so on. The real world does not satisfy these assumptions. They are induced from experience and carry a bias relative to reality—that is inductive bias. The same holds in deep learning. Take neural networks: various architectures, components, and mechanisms often come from inductive bias. In CNNs we assume features have locality—putting neighboring features together makes a solution easier to find; in RNNs we assume each step depends on past computation; attention mechanisms also come from rules induced from human intuition and life experience.

Common Inductive Biases

  • Maximum conditional independence: If a hypothesis can be cast as a Bayesian model, try to maximize conditional independence. This is the bias behind the Naive Bayes classifier.

  • Minimum cross-validation error: When choosing among hypotheses, pick the one with lowest cross-validation error. Though cross-validation may seem unrelated to bias, the no-free-lunch theorem shows it is already biased.

  • Maximum margin: When drawing a boundary between two classes, try to maximize the margin width. This is the bias behind support vector machines—the assumption that different classes are separated by a wide boundary.

  • Minimum description length: When forming a hypothesis, try to minimize its description length. Simpler hypotheses are more likely to be true. See Occam’s razor.

  • Minimum number of features: Unless there is strong evidence a feature is useful, it should be removed. This is the assumption behind feature selection.

  • Nearest neighbor: Assume that in a small region of feature space most samples belong to the same class. For an unknown sample, guess it belongs to the same class as its nearest neighbors. This is the bias behind k-nearest neighbors—the assumption that nearby samples tend to share a class.

Reference links: https://www.zhihu.com/question/264264203/answer/830077823 https://zh.wikipedia.org/wiki/%E6%AD%B8%E7%B4%8D%E5%81%8F%E7%BD%AE