Writing

NLP (3) — Seq2Seq

Encoder-decoder Seq2Seq models, attention mechanisms, and cross-entropy loss for sequence-to-sequence tasks.

  • What is a Seq2Seq network? The Seq2Seq model uses an Encoder-Decoder architecture where the Encoder is an RNNCell (RNN, GRU, LSTM, etc.). A four-layer LSTM structure extracts enough features to improve the decode model.

  • Several Seq2Seq modes 1. Top-student mode image 2. Ordinary cheating image 3. Weak-student cheating image Based on ordinary cheating, look back at the previous step’s answer 4. Bottom-student cheating (attention mechanism) image Highlight the key points in class

  • Application scenarios Any sequence-to-sequence task can use it

  • How is the attention mechanism introduced? image Step one image Step two: image Step three: image

  • Seq2Seq loss function The loss function is cross-entropy. In deep learning, when softmax is used as the classifier, cross-entropy is typically chosen as the loss function.