What Is Positional Encoding
- 1 day ago
- 30 min read
"The dog bit the man" and "The man bit the dog" contain identical words, yet they describe opposite events, and a language model that cannot tell which noun came first will misread both sentences the same way; positional encoding is the mechanism that stops this from happening inside a Transformer, and understanding it is the fastest route to understanding why modern AI can read, write, and reason over long stretches of text at all.
TL;DR
Positional encoding injects sequence-order information into Transformer models, which otherwise treat tokens as an unordered set during self-attention [1].
The original Transformer used fixed sine and cosine functions at different frequencies to build a unique signature vector for every position [1].
Later methods moved position information out of the embeddings and into the attention mechanism itself, including Shaw-style relative representations, Transformer-XL, and T5's relative position bias [2][3][4].
Rotary Position Embedding (RoPE) rotates query and key vectors by an angle tied to position, encoding relative distance directly into attention dot products [5].
ALiBi skips embeddings entirely and adds a distance-based penalty straight to attention scores, which helps some models handle longer sequences than they were trained on [6].
No single method wins universally; the right choice depends on architecture, training length, and whether long-context extrapolation matters more than raw performance at trained lengths.
What Is Positional Encoding
Positional encoding is a technique that adds information about token order to a Transformer model. Because self-attention processes all tokens in parallel, it has no built-in sense of sequence order. Positional encoding, often sine-and-cosine values added to token embeddings, tells the model where each word sits, so word order can shape meaning.
Table of Contents
What Positional Encoding Means
In plain English, positional encoding is a way of telling a model where a word sits in a sentence, so that identical words in a different order are not treated as identical inputs. In technical terms, positional encoding is a set of values, one vector per position, added to or combined with token embeddings before or during attention, so the model's internal representation of a token depends on both what the token is and where it occurs.
Consider "The dog bit the man" versus "The man bit the dog." The vocabulary is the same, the count of each word is the same, and the individual token meanings do not change. What changes is which noun is the subject and which is the object, and that distinction lives entirely in word order. A model that only sees a bag of word embeddings, with no order information, would represent both sentences identically at the token level. Positional encoding exists to close that gap, so the arrangement of tokens becomes part of what the model actually processes, not a detail lost in translation to numbers.
This matters because Transformer-based systems, from translation tools to modern chat assistants, rely on getting subject-object relationships, negation scope, and temporal sequence correct. "Not guilty" and "guilty, not" carry different implications if a model cannot track which word came first. Positional encoding is the architectural piece that makes order-sensitive meaning possible inside an architecture that would otherwise ignore it.
Why Self-Attention Needs an Order Signal
Before Transformers, the dominant sequence models were recurrent neural networks (RNNs) and, to a lesser extent, convolutional neural networks (CNNs) applied to sequences. RNNs process one token at a time, carrying a hidden state forward from step to step; order is baked into the computation because step 5 can only run after step 4 has produced its hidden state. CNNs applied to sequences use sliding filters that look at a local neighborhood of tokens, so relative position within that window is implicit in which filter weights touch which inputs.
The Transformer, introduced by Vaswani and colleagues at Google Brain in 2017, dispensed with both recurrence and convolution in favor of self-attention, which lets every token attend to every other token in a single parallel operation [1]. This parallelism is a major reason Transformers train faster and scale better than RNNs, but it comes at a cost: a standard self-attention layer, before any positional signal is added, is permutation equivariant. That means if you shuffle the order of input tokens, the set of output vectors is shuffled by the same permutation but is otherwise unchanged. The layer treats the input as a set, not a sequence, unless something external tells it how the items were ordered.
It is more precise to describe this as content-only self-attention lacking an explicit order signal, rather than to say attention "cannot understand order" in any sense. Once positional information is present, whether added to embeddings, injected as a bias in the attention scores, or encoded through rotation, order becomes something the model can and does learn to use. Architectural details also matter: causal masking, used in decoder-only models like GPT-style architectures, restricts each token to attending only to earlier tokens. That restriction alone gives the model indirect cues about relative position, since the number of tokens a given position can attend to grows with its index [8]. This nuance, sometimes called the NoPE (No Positional Encoding) finding, is discussed later in this guide; it does not mean positional mechanisms are unnecessary, only that causal structure can leak some order information on its own.
A simple way to see the underlying problem is to picture a token matrix: three tokens, "cat," "sat," "mat," represented as three embedding vectors stacked in a table. Self-attention computes attention scores from queries, keys, and values derived from that table. If you swap the rows, the attention computation still runs, and it produces reordered outputs corresponding to the new row order, but nothing in the raw computation "knows" that row 2 originally came after row 1 rather than before it. The model needs a signal, layered on top, that distinguishes "sat" appearing at position 2 in "cat sat mat" from "sat" appearing at position 1 in "sat cat mat." That signal is positional encoding.
Encoder and decoder attention differ slightly in how order matters. Encoder self-attention (as in the original Transformer's source-side encoder, or in BERT-style models) is typically bidirectional; every position can see every other position, so positional encoding is the only thing distinguishing "before" from "after." Decoder self-attention adds a causal mask on top of positional information, which is why decoder-only generative models can partly infer order from the mask alone, even though they still benefit from explicit positional signals during training [8].
Token Embeddings and Position Vectors
A token embedding is a dense vector that represents what a token means, learned so that tokens used in similar contexts end up with similar vectors, sitting near one another in a high-dimensional embedding space. Token embeddings, on their own, contain no information about where in the sequence a token appears; the same embedding for "bank" is used whether the word shows up first or last in the sentence.
Position information contributes a separate signal: a value, usually also a vector, that depends only on the index of the token in the sequence, not on which token occupies that index. The original Transformer combines the two through addition:
$$x_{pos} = e_{token} + p_{pos}$$
Here, $e_{token}$ is the token embedding for the word at this step, $p_{pos}$ is the positional encoding vector for this position, and $x_{pos}$ is the combined vector that actually enters the first Transformer layer. Both $e_{token}$ and $p_{pos}$ have the same dimensionality, $d_{model}$, which is what makes elementwise addition possible in the first place [1].
Addition is used rather than concatenation mainly for practical reasons. Concatenating a position vector onto a token embedding would either shrink the token embedding's own capacity (if the total dimension is fixed) or grow $d_{model}$ and every downstream weight matrix that touches it, increasing parameter count and compute cost throughout the network. Addition keeps the shape unchanged and lets the model's learned attention and feed-forward weights implicitly separate the "content" and "position" components of the combined vector, since sinusoidal position vectors and token embeddings tend to occupy different regions of the space in practice. Some later architectures move away from addition altogether, injecting relative position bias directly into attention scores instead, which sidesteps the addition-versus-concatenation question by not adding a position vector to the token stream at all [2][3][4].
A tensor-shape walkthrough helps make the mechanics concrete. Suppose a batch contains 32 sequences, each 128 tokens long, with $d_{model} = 512$. Token embeddings form a tensor of shape [32, 128, 512]. A sinusoidal positional encoding table for 128 positions and 512 dimensions has shape [128, 512]. Broadcasting rules in frameworks like PyTorch expand the [128, 512] table across the batch dimension automatically, so the elementwise sum with [32, 128, 512] token embeddings produces a [32, 128, 512] result, with the same positional vector reused for every sequence in the batch (since position 5 means the same thing structurally in every sequence, even though the token occupying that position differs). This shape compatibility is what makes addition simple, and it is one reason $d_{model}$ is chosen once and shared across token and positional representations.
How Sinusoidal Positional Encoding Works
The Original Equations
The original Transformer paper defines positional encoding using alternating sine and cosine functions across dimensions [1]:
$$PE_{(pos,2i)} = \sin\left(\frac{pos}{10000^{2i/d_{model}}}\right)$$
$$PE_{(pos,2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{model}}}\right)$$
Here, $pos$ is the token's position in the sequence, starting at 0; $i$ is an index over dimension pairs, running from 0 up to $d_{model}/2 - 1$; $d_{model}$ is the model's embedding dimension, which is generally chosen to be even so that dimensions pair up cleanly into a sine and a cosine at each frequency; $2i$ refers to the even-indexed dimension in a pair, and $2i+1$ refers to the odd-indexed dimension in that same pair. The constant 10000 is the base used in the original paper to set the range of wavelengths; it is a specific design choice from that paper, not a universal mathematical law, and other bases have been explored in later work without changing the core idea.
Every dimension pair $(2i, 2i+1)$ has its own frequency, determined by $i$. Low values of $i$ produce a denominator close to 1, so the angle grows quickly with $pos$, giving high-frequency oscillation that changes rapidly from one position to the next. High values of $i$ produce a denominator close to $10000$, so the angle grows slowly, giving low-frequency oscillation that changes gradually across many positions. The result is a bank of sine and cosine waves of different wavelengths layered across the dimensions of a single vector.
What the Frequencies Mean
Think of a set of clock hands moving at different speeds: the second hand completes a full circle every minute, the minute hand every hour, the hour hand every twelve hours. Reading all three hands together tells you far more about the exact time than reading just one, because the fast hand disambiguates fine detail while the slow hand disambiguates broad position within a longer cycle. Sinusoidal positional encoding works on a similar principle: high-frequency dimensions distinguish nearby positions from one another, while low-frequency dimensions distinguish broad regions of the sequence, and together the full vector across all dimension pairs forms something like a positional signature.
This is why a single raw position number (just using $pos$ itself as one extra input value) is not typically used: a raw integer scales in a way that is hard for downstream linear layers to use consistently across very different position magnitudes, and it provides no structured way to represent relative offsets. The sinusoidal construction has a useful property instead: for any fixed offset $k$, the positional encoding at position $pos + k$ can be expressed as a linear function of the positional encoding at position $pos$, because of standard angle-addition identities for sine and cosine. This means relative offsets have a comparatively simple relationship to the encoding, which the original paper suggested could make it easier for the model to learn to attend by relative position [1]. It is important not to overstate this: the sinusoidal scheme does not guarantee that every position remains perfectly distinguishable at arbitrary lengths, especially at high position values combined with finite floating-point precision, and it does not, by itself, provide reliable extrapolation to sequence lengths well beyond what was seen in training.
A Worked Numerical Example
Here is a small, self-contained example using $d_{model} = 4$, so there are two frequency pairs: $i = 0$ (dimensions 0 and 1) and $i = 1$ (dimensions 2 and 3). With base 10000, the exponent for $i=0$ is $2(0)/4 = 0$, giving a denominator of $10000^0 = 1$; the exponent for $i=1$ is $2(1)/4 = 0.5$, giving a denominator of $10000^{0.5} = 100$. So dimensions 0 and 1 use angle $pos$, and dimensions 2 and 3 use angle $pos/100$. All values below are rounded to four decimal places.
Position | Dim 0: sin(pos) | Dim 1: cos(pos) | Dim 2: sin(pos/100) | Dim 3: cos(pos/100) |
0 | 0.0000 | 1.0000 | 0.0000 | 1.0000 |
1 | 0.8415 | 0.5403 | 0.0100 | 1.0000 |
2 | 0.9093 | -0.4161 | 0.0200 | 0.9998 |
3 | 0.1411 | -0.9900 | 0.0300 | 0.9996 |
Dimensions 0 and 1 swing widely from one position to the next, since they use the fastest frequency in this tiny example, while dimensions 2 and 3 barely move, since their frequency is much slower. Now suppose a token embedding at position 2 is $e = [0.10, 0.20, -0.10, 0.05]$. Adding the position-2 row from the table gives the combined input vector:
$$x_2 = [0.10 + 0.9093,\ 0.20 - 0.4161,\ -0.10 + 0.0200,\ 0.05 + 0.9998] = [1.0093,\ -0.2161,\ -0.0800,\ 1.0498]$$
This combined vector, not the raw token embedding alone, is what the first Transformer layer actually receives for this token.
How Positional Encoding Enters a Transformer
At a high level, the flow looks like this:
Tokens → Token Embeddings → Add/Inject Position Information → Transformer LayersFor fixed sinusoidal encoding and learned absolute embeddings, the "add" step happens once, before the first Transformer layer, and the resulting combined vectors flow through every subsequent layer unchanged in terms of positional content. For relative methods, T5-style bias, and RoPE, position information is instead injected inside the attention computation at every layer, either by biasing attention scores directly or by rotating query and key vectors before the dot product is taken. ALiBi is a bias-injection method as well, applied at every attention layer rather than once at the input.
Major Types of Positional Encoding
Fixed Sinusoidal Encoding
Fixed sinusoidal encoding is generated by a formula, not learned during training; the same sine and cosine table is computed once and reused for every training example and every inference call [1]. It adds no trainable parameters to the model, which keeps parameter count and memory footprint low. Its main advantages are simplicity, determinism, and the theoretical ability to compute a positional vector for any position, even one never seen during training, since the formula itself has no upper bound. In practice, however, this does not translate into strong empirical length extrapolation on its own; models trained with sinusoidal encoding still tend to degrade when evaluated well past their training length, since the rest of the network's weights were tuned on a particular range of positional values. Historically, sinusoidal encoding was the default in the original Transformer and in several early encoder-decoder systems, though many later architectures moved to alternatives.
Learned Absolute Embeddings
Learned absolute positional embeddings replace the fixed formula with a lookup table: a matrix of shape [max_position, d_model], initialized randomly and updated by gradient descent along with every other model parameter in the network. Position 0 gets its own trainable row, position 1 gets another, and so on, up to some maximum configured length. Models such as GPT-2 and BERT popularized this approach.
The clear limitation is the maximum-position cap: since the table has a fixed number of rows, the model has no principled way to represent a position beyond that cap. Feeding a longer sequence either requires truncation, a separate extension technique, or simply produces an out-of-distribution lookup for positions the table was never trained on. Parameter cost scales with max_position × d_model, which is modest compared to the rest of a large model but is not zero, unlike fixed sinusoidal encoding. The advantage is flexibility: because the vectors are learned rather than fixed by a formula, the model can shape each position's representation to whatever pattern is empirically useful, rather than being constrained to a sine-and-cosine basis. This flexibility trades away the fixed method's ability to compute a coherent vector for an arbitrary, previously unseen position; learned tables have no defined behavior beyond their trained range at all, whereas fixed sinusoidal encoding at least produces a well-formed (if not necessarily useful) vector for any position.
Relative Position Representations
Shaw, Uszkoreit, and Vaswani proposed relative position representations for self-attention, arguing that what matters for many tasks is not a token's absolute index but its distance from the token it is attending to [2]. Rather than adding a position vector to the token embedding, this approach introduces learnable vectors representing relative offsets (such as "two positions to the left" or "three positions to the right") and incorporates them directly into the attention score and value computation. Offsets are typically clipped at some maximum distance, so very distant relationships collapse into a single "far away" representation rather than requiring an unbounded number of offset vectors.
This clipping is both a strength and a limitation: it keeps the parameter count and computation bounded regardless of sequence length, and it reflects a reasonable intuition that very long-range relationships often matter less precisely than short-range ones, but it also means the model cannot distinguish between, say, a token 50 positions away and one 500 positions away if both fall outside the clipping range.
Transformer-XL
Transformer-XL, from Dai and colleagues, introduced a positional formulation designed to work alongside a segment-level recurrence mechanism, letting the model carry hidden states across fixed-length segments so that a new segment can still attend to information from the previous one [3]. Standard absolute positional encoding does not work well here, because the same absolute position index would be reused across different segments, creating ambiguity about how a currently-processed position relates to the memory carried over from an earlier segment. Transformer-XL's positional scheme instead injects relative position information into the attention computation, so that what matters is the offset between the querying token and the token being attended to, regardless of which segment either one falls in. This design, combined with the recurrence mechanism, lets Transformer-XL model dependencies substantially longer than a single fixed-length segment.
T5 Relative Position Bias
T5, from Raffel and colleagues at Google, uses a different relative approach: a small set of learned scalar biases, one per relative-distance "bucket," added directly to the attention logits before the softmax [4]. Rather than a full vector per offset, T5 buckets relative distances (grouping many similar-sized distances into a single bucket) using a scheme that gives fine-grained resolution to nearby positions and coarser resolution to distant ones. Each attention head can also learn its own set of bucket biases, giving different heads different sensitivity to distance. This is a distinct mechanism from Shaw-style relative representations: T5's bias is a scalar added to attention scores rather than a vector combined with key or value representations, and its bucketing scheme is specific to this architecture rather than a generic clipped-offset table.
Rotary Position Embedding
Rotary Position Embedding (RoPE), introduced by Su and colleagues, takes a geometric approach: rather than adding anything to the token embedding, RoPE rotates the query and key vectors by an angle that depends on their position before the attention dot product is computed [5]. Each pair of dimensions in the query and key vectors is treated as a two-dimensional point, and that point is rotated by a position-dependent angle using a standard rotation matrix:
$$R(\theta) = \begin{bmatrix} \cos\theta & -\sin\theta \ \sin\theta & \cos\theta \end{bmatrix}$$
Here, $\theta$ is the rotation angle assigned to a given position and dimension pair (following a frequency schedule similar in spirit to the original sinusoidal scheme), and $R(\theta)$ is the 2x2 rotation matrix applied to that pair of coordinates. Applying this rotation independently to many dimension pairs, each with its own frequency, produces the full rotated query or key vector.
The useful property emerges when a rotated query at position $m$ is dotted with a rotated key at position $n$: due to how rotation matrices combine, the resulting dot product depends only on the relative offset $m - n$, not on the absolute positions $m$ and $n$ individually. This means RoPE encodes absolute position through the rotation applied to each vector, while the attention mechanism ends up expressing a genuinely relative signal in practice, which is why RoPE is best described as inducing relative-position structure in attention rather than simply adding another positional vector alongside the token embedding [5]. RoPE has become the default positional method in many modern large language models because it integrates cleanly with the standard attention computation and empirically performs well at moderate sequence lengths. Its limitation is that, like other methods, it does not automatically guarantee strong behavior far beyond its trained length; a range of scaling techniques, covered in the next major section, exist specifically to extend RoPE-based models to longer contexts.
ALiBi
Attention with Linear Biases (ALiBi), from Press, Smith, and Lewis, takes yet another route: it adds no positional vectors to embeddings and applies no rotation to queries and keys [6]. Instead, ALiBi subtracts a penalty from the raw attention logits that grows linearly with the distance between the query and key positions, before the softmax is applied. Each attention head is assigned its own fixed slope for this penalty, so some heads penalize distant tokens heavily (favoring very local attention) while other heads penalize them lightly (allowing broader context), and these slopes are set by a fixed schedule rather than learned.
Because the penalty is a simple function of distance with no learned position-specific parameters, ALiBi adds no extra parameters at all. Its motivation was explicitly length extrapolation: Press and colleagues showed that models trained with ALiBi at a given sequence length could be evaluated on notably longer sequences with smaller increases in perplexity than several alternative positional methods tested in the same study, while also training somewhat faster and using less memory during their reported experiments [6]. ALiBi differs from additive encodings by never touching the embeddings, from Shaw-style and T5-style relative embeddings by using a fixed formula rather than learned per-distance values, and from RoPE by biasing scores additively rather than rotating vectors multiplicatively.
Transformers Without Explicit Positional Encoding
A body of research, most notably from Haviv and colleagues, has examined what happens when a causal (decoder-only) Transformer receives no explicit positional signal at all, often called the NoPE setting [8]. Their findings show that such models remain competitive with models trained using explicit positional encoding on the tasks and scales they tested, and that probing experiments can recover an implicit notion of absolute position from the network's internal representations, even though nothing in the architecture was designed to represent position directly. The proposed explanation is that causal masking itself, by restricting each token to attend only to earlier tokens, gives the network indirect information about how many predecessors a given position has, which the network can learn to exploit as an implicit position signal.
Note: This finding is a meaningful nuance about causal architectures specifically, not evidence that positional mechanisms are universally unnecessary. It does not apply to bidirectional encoders without causal masking, where no equivalent implicit signal exists, and the experimental scope of NoPE studies does not establish that every causal architecture at every scale and task will behave identically. Explicit positional methods remain the default in the large majority of production Transformer models.
Positional Encoding and Long Context Windows
Training length and inference length are not the same thing, and the gap between them is where most long-context engineering happens. A model trained with sequences capped at, say, 4,096 tokens has only ever seen positional signals (whatever the method) within that range. Asking it to process a 32,000-token input means asking its positional mechanism, and every downstream weight tuned around it, to behave sensibly for values it never encountered during training. This is the distinction between interpolation, where positions are mapped into the range the model already saw, and extrapolation, which requires the model to behave correctly beyond that trained range.
Distribution shift is the core problem: a positional value the model never trained on can produce attention patterns that behave erratically, showing up as degraded performance the further a task pushes past the trained length, even when raw architectural capacity would seem to allow longer inputs. Several techniques have been developed specifically to manage this gap for RoPE-based models. Position Interpolation, introduced by Chen and colleagues, rescales position indices so that a longer sequence's positions are compressed to fall within the range the model originally trained on, effectively turning an extrapolation problem into an interpolation problem, followed by a short period of fine-tuning [9]. YaRN, from Peng and colleagues, refines this idea by scaling different RoPE frequency bands differently rather than applying one uniform scale factor, based on the observation that high- and low-frequency dimensions respond differently to naive interpolation, and reports strong context-extension results with comparatively little additional fine-tuning [10]. XPos, introduced in "A Length-Extrapolatable Transformer" by Sun and colleagues, modifies the RoPE formulation itself by adding a decay term to the rotation, aiming to improve extrapolation behavior directly rather than through a separate rescaling step applied after training [11].
Warning: A configured maximum context length, theoretical architectural capacity, and effective context utilization are three different concepts. A model might be configured to accept 128,000 tokens, yet use information from the middle of that window less reliably than information near the beginning or end, a pattern observed across a range of long-context evaluations. Usable context also depends on factors well beyond the positional method: the training data distribution (whether long documents were common during pretraining), the attention architecture and memory budget at inference time, retrieval and augmentation strategies layered on top (as in retrieval-augmented generation systems), and how a task's evaluation methodology probes for information anywhere in the input versus only near the edges. Positional encoding shapes what is architecturally possible, but it does not, by itself, guarantee that a model will use every part of a long context equally well.
Comparison of Positional Encoding Methods
Method | Where Position Enters | Fixed or Learned | Absolute or Relative | Extra Parameters | Typical Extrapolation | Main Strengths | Main Limitations | Common Use Cases |
Fixed sinusoidal [1] | Added to input embeddings | Fixed | Absolute (with relative structure available) | None | Weak in practice | Simple, no added parameters, defined for any position | Limited real extrapolation, no learned adaptation | Original Transformer, early encoder-decoder MT |
Learned absolute [1] | Added to input embeddings | Learned | Absolute | max_pos × d_model | None beyond trained max | Flexible, task-adapted | Hard cap on sequence length | BERT, GPT-2-era models |
Shaw relative [2] | Inside attention (keys/values) | Learned | Relative (clipped) | Small, clipped-offset table | Limited beyond clip range | Captures local distance well | Distant offsets collapse together | Machine translation, encoder tasks |
Transformer-XL [3] | Inside attention, with segment recurrence | Learned | Relative | Moderate | Better than fixed-segment models | Handles cross-segment memory | Added architectural complexity | Long-document language modeling |
T5 relative bias [4] | Added to attention logits | Learned | Relative (bucketed) | Small, per-head scalars | Moderate | Efficient, per-head distance sensitivity | Coarse resolution for distant tokens | T5 and text-to-text models |
RoPE [5] | Rotates queries and keys | Fixed formula, no trainable position parameters | Relative (via rotation of absolute-position vectors) | None | Moderate, needs scaling for long context | Integrates smoothly with attention, strong empirical results | Needs extension techniques for long context | Most modern large language models |
ALiBi [6] | Added to attention logits | Fixed slopes | Relative (distance penalty) | None | Reported strong extrapolation in original study | No embedding parameters, efficient | Fixed recency bias may not suit every task | Long-context language modeling research |
NoPE [8] | Implicit, via causal mask only | None (no explicit encoding) | Implicit | None | Competitive in-distribution, generalization findings task- and scale-dependent | No positional parameters at all | Not established for bidirectional or all settings | Research settings, causal LMs specifically |
PyTorch Implementation
The following implementation builds the original sinusoidal positional encoding described in the equations above, precomputes the table once, and stores it as a buffer rather than a trainable model weight, since fixed sinusoidal values should not receive gradient updates.
import math
import torch
import torch.nn as nn
class SinusoidalPositionalEncoding(nn.Module):
"""
Fixed sinusoidal positional encoding, as described by Vaswani et al. (2017).
Expects input of shape [batch_size, sequence_length, d_model].
"""
def __init__(self, d_model: int, max_len: int = 5000, base: float = 10000.0):
super().__init__()
if d_model % 2 != 0:
raise ValueError("d_model must be even for paired sine/cosine dimensions.")
# Position indices: shape [max_len, 1]
position = torch.arange(max_len, dtype=torch.float32).unsqueeze(1)
# Frequency term for each dimension pair: shape [d_model / 2]
div_term = torch.exp(
torch.arange(0, d_model, 2, dtype=torch.float32)
* (-math.log(base) / d_model)
)
pe = torch.zeros(max_len, d_model)
pe[:, 0::2] = torch.sin(position * div_term) # even dimensions
pe[:, 1::2] = torch.cos(position * div_term) # odd dimensions
# Add a batch dimension: [1, max_len, d_model]
pe = pe.unsqueeze(0)
# register_buffer: moves with .to(device), saved in state_dict,
# but excluded from optimizer updates (no gradients tracked).
self.register_buffer("pe", pe, persistent=False)
def forward(self, x: torch.Tensor) -> torch.Tensor:
# x: [batch_size, sequence_length, d_model]
seq_len = x.size(1)
if seq_len > self.pe.size(1):
raise ValueError(
f"Sequence length {seq_len} exceeds max_len {self.pe.size(1)} "
"used to build the positional encoding table."
)
# Broadcasting: [1, seq_len, d_model] + [batch, seq_len, d_model]
return x + self.pe[:, :seq_len, :]
# Usage example
d_model = 512
batch_size = 32
seq_len = 128
token_embeddings = torch.randn(batch_size, seq_len, d_model)
pos_encoder = SinusoidalPositionalEncoding(d_model=d_model, max_len=1024)
output = pos_encoder(token_embeddings)
print(f"Input shape: {token_embeddings.shape}") # [32, 128, 512]
print(f"Output shape: {output.shape}") # [32, 128, 512]Walking through the important lines: div_term computes the frequency for each dimension pair in log-space for numerical stability, equivalent to $1 / 10000^{2i/d_{model}}$. The slicing pe[:, 0::2] fills every even-indexed dimension with sine values, and pe[:, 1::2] fills every odd-indexed dimension with cosine values, directly mirroring the two equations from the original paper. Registering pe with register_buffer and persistent=False means the tensor moves automatically when the module is sent to a GPU with .to(device), but it is not included in model.parameters(), so no optimizer will attempt to update it and no gradient will ever flow into it; this is exactly right for a table computed by a fixed formula, since there is nothing to learn.
Broadcasting is what allows a [1, seq_len, d_model] positional table to be added directly to a [batch_size, seq_len, d_model] embedding tensor: PyTorch expands the size-1 batch dimension to match, without copying memory. For a sequence-first layout ([sequence_length, batch_size, d_model], used by some older codebases), the positional table would instead be shaped [seq_len, 1, d_model], moving the broadcastable dimension to the middle rather than the front. The max_len parameter sets a hard ceiling: any sequence longer than the precomputed table raises an error rather than silently producing incorrect or truncated positional values, which matters because silent truncation of a long input's positional signal can be a subtle source of bugs.
How to Choose a Positional Method
There is no single correct choice; the right method depends heavily on what you are building.
Teaching or reproducing the original Transformer: fixed sinusoidal encoding is the historically accurate and pedagogically clearest choice [1].
Training a new model from scratch with a fixed, modest maximum length: learned absolute embeddings are simple and often perform well within that fixed range.
Building a long-context autoregressive model from the ground up: RoPE is the most common modern default, since most current large language model families use it and tooling support is mature [5].
Extending an existing RoPE-based model to longer sequences without retraining from scratch: Position Interpolation or YaRN are the established techniques for stretching an already-trained RoPE model's effective range [9][10].
Prioritizing simplicity and minimal parameter overhead in research code: ALiBi or fixed sinusoidal encoding avoid extra learned components entirely [6].
Prioritizing extrapolation behavior specifically as a research question: ALiBi, XPos, and NoPE studies are the relevant literature to study side by side [6][8][11].
Handling two-dimensional inputs, such as images: none of the one-dimensional schemes above apply directly; 2D-adapted variants are needed, discussed in the next section.
Reproducing a specific published architecture exactly: match whatever that paper used; positional method choices are rarely interchangeable without retraining, since a network's other weights are tuned around a specific positional signal.
Tip: None of this is a universal prescription. A team fine-tuning an existing base model, whether through an AI API or by adapting open weights, inherits whatever positional method that base model already uses; the practical decision in that setting is usually about scaling techniques for longer context rather than swapping the underlying mechanism.
Applications Beyond Text
One-dimensional token order is not the only kind of positional structure a model might need to represent. Vision Transformers (ViTs) split an image into a grid of patches and need to represent two-dimensional position, row and column, rather than a single linear index; common approaches either learn a 2D positional embedding table or adapt rotary-style encodings to two axes simultaneously. Audio and speech models, which process sequences of frames over time, often reuse one-dimensional positional schemes similar to text, sometimes combined with relative timing information specific to acoustic structure. Time-series models face a related but distinct challenge, since real-world timestamps can be irregularly spaced, which sometimes calls for encodings that reflect elapsed time rather than a simple integer step count.
Multimodal models, which combine text, images, and sometimes audio in a single architecture, often need to reconcile different positional schemes across modalities, assigning position within each modality's own structure while also representing how the modalities relate to one another; this is an active area of ongoing research rather than a fully settled problem. Graph-structured data introduces yet another variant: since a graph has no single canonical linear order, positional schemes for graph Transformers instead often encode structural information, such as shortest-path distance between nodes, rather than a sequential index. Across all of these domains, the underlying goal is the same one this guide has focused on for text: giving an otherwise order-agnostic attention mechanism a way to represent the structure that actually exists in the data.
Common Misconceptions
"Attention automatically knows token order." Content-only self-attention is permutation equivariant; order awareness comes from positional encoding or, in causal architectures, partly from the causal mask itself [8].
"Position and meaning are the same thing." Position tells the model where a token sits; the token embedding itself carries what the token means. They are combined, not merged into one concept.
"Positional encoding is always learned." Fixed sinusoidal encoding and ALiBi use no learned positional parameters at all; only some methods, like learned absolute embeddings and T5's bias, involve training [1][6].
"All positional encodings are added to token embeddings." RoPE rotates queries and keys, and ALiBi biases attention logits directly; neither adds a vector to the token embedding [5][6].
"Sinusoidal encoding extrapolates perfectly." The formula is defined for any position mathematically, but empirical performance still degrades well past the trained length, since the rest of the network was never tuned for those values [1].
"RoPE solves long context by itself." RoPE provides a favorable structure for relative position, but models still need specific scaling techniques, such as interpolation or YaRN, to perform well far beyond their trained length [9][10].
"Relative methods represent no absolute information." RoPE, for instance, encodes absolute position through rotation angles even though the resulting attention behavior is relative; absolute and relative signals are not mutually exclusive [5].
"Every Transformer must use an explicit positional encoding." NoPE research shows causal models can learn competitive performance without one, though this finding is scoped to causal architectures and specific experimental settings, not a universal claim [8].
"Maximum context length and effective context use are identical." A model configured for a large maximum length can still use tokens in the middle of that window less reliably than tokens near the edges.
"A higher position number can simply be supplied to any model." Feeding a position index beyond a model's trained range, especially for learned absolute embeddings, produces undefined or degraded behavior rather than a sensible extrapolation.
Final Conceptual Synthesis
Positional information contributes exactly one thing that content alone cannot: a signal for where a token sits relative to the rest of the sequence, which is what turns an unordered bag of vectors into a genuine sequence a model can reason over. Multiple mechanisms exist because that single goal can be pursued at different points in the architecture, added to embeddings once at the input, injected as a bias inside every attention layer, or encoded as a rotation applied to queries and keys, and each choice carries different trade-offs around parameter cost, extrapolation behavior, and how cleanly it integrates with the rest of the model.
The best method depends on the architecture it sits inside and the task it needs to serve: a model built for a fixed, moderate context length has different requirements than one built to scale toward very long documents or codebases. This is precisely why positional encoding connects so directly to long-context modeling; scaling a model's usable context is rarely just a matter of raising a configuration number, and it typically involves rethinking how the positional mechanism itself behaves at ranges the model never directly trained on.
FAQ
What is positional encoding in simple terms?
Positional encoding is information added to a Transformer model that tells it where each token sits in a sequence. Without it, the model would treat "dog bites man" and "man bites dog" as the same set of words in no particular order.
Why do Transformers need positional encoding at all?
Self-attention processes all tokens in parallel and is permutation equivariant on its own, meaning it has no built-in sense of order [1]. Positional encoding, or an equivalent mechanism like a causal mask's implicit signal, is what supplies that missing order information.
What is the difference between positional encoding and positional embedding?
"Positional encoding" is often used as a broad umbrella term for any technique that injects position information into a model. "Positional embedding" more often refers specifically to a learned lookup table of position vectors, though usage varies across papers and libraries.
What do the sine and cosine functions in positional encoding actually do?
They generate a bank of waves at different frequencies across the embedding dimensions, so each position gets a distinctive combination of values [1]. Fast-changing dimensions distinguish nearby positions, while slow-changing dimensions distinguish broader regions of the sequence.
What is the difference between absolute and relative positional encoding?
Absolute methods, like fixed sinusoidal encoding or learned embeddings, assign a distinct signal to each specific index in the sequence. Relative methods, like Shaw-style representations or T5's bias, instead represent the distance between two tokens, regardless of their exact positions [2][4].
How do learned absolute positional embeddings work?
They use a trainable lookup table with one row per position, up to a fixed maximum length, updated by gradient descent along with the rest of the model. Their main limitation is that they have no defined behavior for positions beyond that trained maximum.
How does RoPE encode position?
RoPE rotates pairs of dimensions in the query and key vectors by an angle tied to their position, using a rotation matrix [5]. When rotated queries and keys are dotted together in attention, the result depends on their relative offset rather than their absolute positions.
What is ALiBi and how is it different from RoPE?
ALiBi adds a distance-based penalty directly to attention scores, using fixed per-head slopes, rather than rotating vectors or adding embeddings [6]. It was designed specifically to help models trained at one length perform better when evaluated on longer sequences.
Can a Transformer work without any explicit positional encoding?
Research on NoPE shows that causal Transformers can remain competitive without explicit positional encoding, partly because the causal mask itself leaks some order information [8]. This finding is scoped to causal architectures and does not establish that positional mechanisms are unnecessary in bidirectional models or at every scale.
Does positional encoding guarantee good performance on long contexts?
No. Usable context length also depends on training data, attention architecture, memory constraints, and evaluation methodology, not positional encoding alone. Techniques like Position Interpolation and YaRN exist specifically because positional methods need extra help to extend well beyond their trained range [9][10].
What is the maximum sequence length a positional encoding method supports?
It depends on the method. Fixed sinusoidal encoding and ALiBi are mathematically defined for any position, though performance still degrades past the trained range; learned absolute embeddings have a hard cap set by the size of their lookup table.
Why is positional information added instead of concatenated to token embeddings?
Addition keeps the vector dimension unchanged, avoiding extra parameters and larger downstream weight matrices, since both vectors share the same $d_{model}$ [1]. Concatenation would require either shrinking the token embedding's own capacity or increasing the model's dimensionality throughout.
Is positional encoding used outside of text-based Transformers?
Yes. Vision Transformers use two-dimensional positional schemes for image patches, audio and time-series models often adapt one-dimensional schemes, and graph Transformers frequently encode structural distance between nodes instead of a linear sequence index.
How do I choose the right positional encoding method for my project?
The choice depends on your architecture and goals: fixed sinusoidal or learned embeddings suit fixed-length, from-scratch training, while RoPE is the common default for modern long-context language models [5]. If you are extending an existing RoPE-based model, Position Interpolation or YaRN are the standard tools for that specific job [9][10].
What is T5's relative position bias?
T5 adds a small set of learned scalar biases to attention logits, based on which distance bucket a token pair falls into, with finer resolution for nearby tokens and coarser resolution for distant ones [4]. It differs from Shaw-style relative representations, which use vectors combined with keys and values rather than scalar biases on the logits.
Key Takeaways
Positional encoding gives Transformer models a way to represent sequence order, which content-only self-attention does not provide on its own [1].
The original sinusoidal formulation builds a unique signature per position from sine and cosine waves at different frequencies, without adding any trainable parameters [1].
Learned absolute embeddings, relative representations, T5's bucketed bias, RoPE, and ALiBi each inject positional information at a different point in the architecture, with different trade-offs [1][2][4][5][6].
RoPE has become the common default in modern large language models because it integrates cleanly with attention and produces a genuinely relative signal from rotated absolute-position vectors [5].
NoPE research shows causal masking alone can leak some order information, but this does not make explicit positional methods unnecessary in general [8].
Long-context performance depends on more than the positional method; training length, architecture, and evaluation methodology all play a role, and techniques like Position Interpolation and YaRN exist specifically to stretch trained models further [9][10].
No single positional method is universally best; the right choice depends on the architecture, training length, and whether extrapolation is a priority.
Actionable Next Steps
If you are implementing a Transformer from scratch for learning purposes, start with the fixed sinusoidal formula from the original paper before exploring alternatives, since it has no trainable parameters to debug [1].
Run the PyTorch implementation in this guide on a small example and print the resulting shapes to confirm your broadcasting logic before scaling up to a full model.
If you are fine-tuning or extending an existing model, check which positional method it already uses before choosing a context-extension technique, since Position Interpolation and YaRN are specific to RoPE-based architectures [5][9][10].
Read the original papers for RoPE and ALiBi directly if you plan to modify either mechanism, since their attention-level integration details matter for correct implementation [5][6].
If working with images, audio, or graph-structured data, look specifically for positional schemes adapted to that modality rather than reusing one-dimensional text methods unmodified.
Before claiming a model handles "long context," test it with information placed at the middle of a long input, not just at the start or end, to check for uneven effective context use.
Glossary
Absolute position: A token's fixed index within a sequence, such as "third token," independent of any other token.
ALiBi (Attention with Linear Biases): A method that subtracts a distance-based penalty from attention logits using fixed per-head slopes, rather than adding position vectors to embeddings [6].
Attention logit: The raw, pre-softmax score computed between a query and a key in an attention layer.
Causal mask: A restriction in decoder-style attention that prevents a token from attending to tokens that come after it in the sequence.
Context window: The span of tokens a model can process or attend to at one time.
$d_{model}$: The dimensionality of the vectors used throughout a Transformer, shared by token embeddings and positional encodings.
Embedding: A dense vector representation of a token, learned so that similar tokens end up close together in embedding space.
Extrapolation: Model behavior on inputs, such as sequence positions, that fall outside the range seen during training.
Interpolation: Mapping new inputs, such as positions, into a range the model already saw during training.
Length extrapolation: A model's ability to maintain performance when evaluated on sequences longer than those used in training.
NoPE: Short for "No Positional Encoding," referring to causal Transformers trained without any explicit positional mechanism [8].
Permutation equivariance: A property where shuffling the input to a function shuffles the output the same way, meaning the function itself has no inherent sense of order.
Position: A token's location within a sequence, expressed as an index.
Positional encoding: Any mechanism, formula-based or learned, that injects sequence-order information into a Transformer model.
Positional embedding: A learned vector representing a specific position, typically stored in a trainable lookup table.
Relative position: The distance or offset between two tokens, rather than either token's absolute index.
RoPE (Rotary Position Embedding): A method that rotates query and key vectors by a position-dependent angle, producing attention scores that depend on relative offset [5].
Self-attention: The mechanism that lets each token in a sequence weigh and combine information from every other token.
Sequence: An ordered list of tokens, where the order itself can carry meaning.
Sinusoidal encoding: Positional encoding built from sine and cosine functions at varying frequencies, as used in the original Transformer [1].
T5 relative position bias: A set of learned scalar biases added to attention logits based on bucketed relative distance, used in the T5 architecture [4].
Token: A unit of input, often a word or subword, that a model processes as a discrete step.
Transformer: A neural network architecture built around self-attention, introduced by Vaswani and colleagues [1].
Transformer-XL: A Transformer variant combining segment-level recurrence with a relative positional scheme suited to that recurrence [3].
YaRN: A technique for extending a RoPE-based model's usable context length by scaling different frequency bands differently [10].
Sources & References
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention Is All You Need. arXiv:1706.03762. https://arxiv.org/abs/1706.03762
Shaw, P., Uszkoreit, J., & Vaswani, A. (2018). Self-Attention with Relative Position Representations. arXiv:1803.02155. https://arxiv.org/abs/1803.02155
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., & Salakhutdinov, R. (2019). Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context. arXiv:1901.02860. https://arxiv.org/abs/1901.02860
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv:1910.10683. https://arxiv.org/abs/1910.10683
Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., & Liu, Y. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv:2104.09864. https://arxiv.org/abs/2104.09864
Press, O., Smith, N. A., & Lewis, M. (2021). Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. arXiv:2108.12409. https://arxiv.org/abs/2108.12409
Dufter, P., Schmitt, M., & Schütze, H. (2022). Position Information in Transformers: An Overview. Computational Linguistics, 48(3), 733–763. https://aclanthology.org/2022.cl-3.7/
Haviv, A., Ram, O., Press, O., Izsak, P., & Levy, O. (2022). Transformer Language Models without Positional Encodings Still Learn Positional Information. arXiv:2203.16634. https://arxiv.org/abs/2203.16634
Chen, S., Wong, S., Chen, L., & Tian, Y. (2023). Extending Context Window of Large Language Models via Positional Interpolation. arXiv:2306.15595. https://arxiv.org/abs/2306.15595
Peng, B., Quesnelle, J., Fan, H., & Shippole, E. (2023). YaRN: Efficient Context Window Extension of Large Language Models. arXiv:2309.00071. https://arxiv.org/abs/2309.00071
Sun, Y., Dong, L., Patra, B., Ma, S., Huang, S., Benhaim, A., Chaudhary, V., Song, X., & Wei, F. (2022). A Length-Extrapolatable Transformer. arXiv:2212.10554. https://arxiv.org/abs/2212.10554
