Transformer — Attention Mechanism & Architecture

Transformer architecture is a neural network design built around self-attention. Instead of relying mainly on step-by-step processing, it lets each token gather information from other relevant tokens in the sequence.

That is why transformers work well for language and other sequence tasks. If one word depends on another word far away, attention gives the model a direct path between them.

What transformer architecture means

A transformer block does more than apply one formula, but self-attention is the central idea. In self-attention, each token produces three vectors:

• a query, which represents what this token is looking for
• a key, which represents what this token offers for matching
• a value, which is the information that can be passed along

If the token representations are arranged in a matrix $X$, one attention head typically forms

$$Q = XW_Q, \qquad K = XW_K, \qquad V = XW_V$$

where $W_Q$, $W_K$, and $W_V$ are learned matrices.

The standard scaled dot-product attention formula is

$$\operatorname{Attention}(Q, K, V) = \operatorname{softmax}\!\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

Here $d_k$ is the key dimension. The scaling by $\sqrt{d_k}$ helps keep the raw dot-product scores from becoming too large as the dimension grows.

The softmax is applied row by row. Each row answers one token's question: "How much attention should I give to the other tokens?"

How self-attention works in plain language

Self-attention does not usually pick one token and ignore the rest. It builds a weighted average of value vectors.

If one token strongly matches another token's key, that token gets a larger weight. If the match is weak, the weight is smaller. The output for one token is therefore a context-aware mixture of information from the sequence.

This helps with subject-verb agreement, pronoun reference, and other long-range relationships. The model does not need information to travel through many intermediate steps before it becomes available.

Worked self-attention example

Take one query and two candidate tokens in a single attention head. To keep the arithmetic simple, use $d_k = 1$.

Suppose the current token has query

$$q = [2]$$

and the two candidate tokens have

$$k_1 = [2], \qquad k_2 = [1]$$

with values

$$v_1 = [10], \qquad v_2 = [4].$$

The raw attention scores are the dot products:

$$qk_1^T = 4, \qquad qk_2^T = 2.$$

Because $d_k = 1$, the scaling factor is $\sqrt{1} = 1$, so the scaled scores are still $4$ and $2$.

Now apply softmax to those two scores:

$$\alpha_1 = \frac{e^4}{e^4 + e^2} \approx 0.881, \qquad \alpha_2 = \frac{e^2}{e^4 + e^2} \approx 0.119.$$

The attention output is the weighted combination

$$\alpha_1 v_1 + \alpha_2 v_2 = 0.881(10) + 0.119(4) \approx 9.29.$$

The key idea is simple: the output lands closer to $v_1$ because the query matched $k_1$ more strongly than $k_2$.

This is the basic pattern inside a much larger model. Real transformers do this in higher dimensions and across many tokens at once, but the arithmetic idea is the same.

What else is inside a transformer block

A transformer is more than one attention formula. A standard block usually contains:

• multi-head attention, so the model can learn several kinds of relationships at once
• a position-wise feedforward network, which transforms each token representation after attention
• residual connections, which help preserve and refine information across layers
• layer normalization, which helps stabilize training

In the original transformer architecture for sequence-to-sequence tasks, the model had an encoder stack and a decoder stack.

• The encoder uses self-attention over the input sequence.
• The decoder uses masked self-attention so a position cannot look ahead to future output tokens.
• The decoder can also use cross-attention, where queries come from the decoder and keys and values come from the encoder output.

Many modern language models use only the decoder side. The core attention idea is still there, but the overall architecture is specialized for next-token prediction.

Why transformers need positional information

Attention alone is permutation-equivariant with respect to the input tokens. In plain language, if you only apply attention to the same set of token vectors without adding position, the model does not inherently know which token came first.

That is why transformers add positional information, such as learned position embeddings or positional encodings. Without that extra signal, order-sensitive tasks like language would be much harder to model correctly.

Common mistakes about transformer architecture

Thinking attention is the whole transformer

It is the central idea, but the architecture also depends on feedforward layers, residual paths, normalization, and positional information.

Mixing up self-attention and cross-attention

In self-attention, $Q$, $K$, and $V$ come from the same sequence. In cross-attention, they do not all come from the same source.

Forgetting the role of masking

Decoder-only language models need causal masking during training and inference so a token cannot attend to future tokens.

Treating attention weights as a complete explanation

Attention weights can be informative, but they are not a full proof of model reasoning. The final behavior also depends on value vectors, later layers, and nonlinear transformations.

When transformer models are used

Transformers are widely used in language modeling, translation, summarization, code generation, speech, and many vision tasks. They work especially well when relationships across a sequence or set matter more than purely local patterns.

They are not magic for every setting. For very small datasets, strict real-time constraints, or problems where local inductive structure matters most, another architecture can still be a better fit.

Try a similar problem

Take a three-word phrase and focus on one word. Decide which of the other words should get high attention weight and why, then sketch a tiny query-key-value example to match that intuition.

If you want to go one step further, compute one small attention output by hand. That is usually the fastest route from "I know the formula" to "I understand what the architecture is doing."