Machine Learning (5) — Logistic Regression

Preface: This covers classification problems. It follows the linear regression article because logistic regression follows nearly the same steps: write the objective, find the loss function, minimize it, solve for parameters, build the model, and evaluate. A classification algorithm and probability model that returns the probability of belonging to a class; cross-entropy loss; its derivative matches gradient descent. Softmax handles multi-class problems and improves accuracy.

Logistic Regression

• Sigmoid function Loss function builds the model and outputs the probability of the target value to solve binary classification. A smaller loss maximizes probability, so take the logarithm of the loss first: Then use differentiation to find the parameters that minimize the loss ## Softmax Regression Solves multi-class problems Write the probability function: Note: Why use the exponential function? To show how small changes in data cause probability to grow exponentially.

• Write the loss function The first summation is over m samples; the second k sums all sample probabilities where y=j. Why use the logarithm? Because in the 0–1 range, the log changes rapidly.

• Differentiate

• Gradient descent to solve for parameters