LLM Technical Notes (1) — Positional Encoding

1. Why Positional Encoding Is Needed

This goes back to the original Transformer design. In human language, word position and order define grammar and affect meaning. Without capturing order, it is hard to understand a sentence.

In NLP, for any neural architecture, recognizing each word’s position and order between words is critical. Traditional RNNs consider order through autoregressive steps; Transformers use pure self-attention to relate words. Pure self-attention is permutation-invariant—Transformers cannot capture sequence order by attention alone. We need a way to inject order into the architecture; positional encoding fills that role.

2. Absolute vs Relative Positional Encoding

Absolute encoding tells the Transformer each element’s position in the input sequence—like tagging each token with an absolute index. Relative encoding works in the attention mechanism and encodes distance between pairs of elements.

3. Rotary Positional Encoding in Large Language Models

Rotary Position Embedding (RoPE), from Roformer: Enhanced Transformer With Rotary Position Embedding, integrates relative position into self-attention and improves Transformer performance. Popular models like LLaMA and GLM use RoPE. Compared with relative encodings, RoPE extrapolates better and is widely used for relative position in large models.

Note: What is extrapolation in large models?
Extrapolation means train and inference sequence lengths differ, hurting generalization. E.g., if training uses only 512 tokens, inputs longer than 512 at inference may fail. That limits long-text and multi-turn dialogue tasks.

See reference links for details.