Writing

Deep Learning — RNN (1)

RNN basics, forward and backward propagation, gradient issues, and LSTM gates.

Preface: Why RNN when we have BP nets and CNNs? BP and CNN inputs/outputs are independent—they model neuron message passing but don’t remember childhood events. RNN adds “memory.”

RNN Basics

Some outputs depend on prior context. RNN introduces memory; recursion means each step runs the same task but output depends on input and memory—common in NLP.

  • Where is the “loop”? Unlike feedforward nets, input x produces state s, saved and fed to the next step—current information is remembered: image

  • Parameters: U (input→hidden), W (hidden→hidden, memory controller), V (hidden→output) image

  • Forward pass At t=1, U,V,W initialized; s0 usually 0: image Time advances; s1 joins the next prediction: image In general: image f may be tanh, relu, sigmoid; g usually softmax. Memory: W summarizes past inputs for the next step—hidden state h = f(current input + past summary).

  • Backprop Like BP: sum output errors, gradients ∇U,∇V,∇W, gradient descent. Each ot has error et; total E=∑t et; minimize: image Chain rule for W: image U, V similarly. Multiple factors <1 cause vanishing gradients—hence LSTM/GRU with special memory that isn’t wiped instantly.

LSTM

LSTM is RNN with a modified memory cell—information to keep flows on; irrelevant is cut. RNN successes in speech, LM, translation, captioning often use LSTM.

  • Structure Standard RNN module—often one tanh layer image LSTM: four interacting layers image Icons: image Black lines carry vectors; pink circles are pointwise ops; yellow boxes are learned layers; merged lines concatenate; split lines copy to multiple places.

  • Cell state Like a conveyor; runs the chain with little change image

  • Gates Sigmoid + pointwise multiply; sigmoid outputs 0–1 pass fraction; three gates control cell state image

  • Forget gate: what to drop—e.g. gender in LM when a new pronoun appears image

  • Input gate: what to add; sigmoid picks updates; tanh creates candidate Ct image

  • Update: Ct-1 → Ct; multiply old state by ft, add it*Ct image

  • Output gate: sigmoid picks output parts; tanh on state, multiply, output image Forward/backward like RNN; formulas: blog, paper