Transformer — Attention Mechanism & Architecture

To really see what a transformer does, compute one self-attention output by hand. The procedure is fixed: build queries, keys, and values from the tokens, score each query against the keys, scale and softmax those scores, then mix the values by the resulting weights. Run that once and the rest of the architecture becomes context around a step you already understand.

Transformer architecture is a neural network design built around self-attention. Instead of relying mainly on step-by-step processing, it lets each token gather information from other relevant tokens in the sequence. If one word depends on another far away, attention gives the model a direct path between them.

When Self-Attention Applies

Use self-attention whenever a token's meaning depends on other tokens in the same sequence. Each token produces three vectors:

• a query, what this token is looking for
• a key, what this token offers for matching
• a value, the information that can be passed along

With token representations in a matrix $X$, one attention head forms

where $W_Q$, $W_K$, $W_V$ are learned matrices.

The Procedure, Step by Step

1. Start with token vectors and add positional information, because attention alone does not tell the model which token came first.
2. Build queries, keys, and values with learned linear maps; in self-attention they come from the same sequence.
3. Compute attention weights: form scores with $QK^T$, scale by $\sqrt{d_k}$, apply softmax row by row, and combine the value vectors. The standard formula is

4. Finish the block by sending the attention output through the rest of the transformer block — usually multi-head attention, a feedforward network, residual connections, and normalization.
5. Stack blocks for the task: encoder-decoder stacks for sequence-to-sequence work, decoder-only stacks for next-token prediction.

In plain language, self-attention does not pick one token and ignore the rest — it builds a weighted average of value vectors. A strong query-key match earns a larger weight; a weak one earns less. Each token's output is a context-aware mixture, which is what helps with subject-verb agreement, pronoun reference, and other long-range relationships. The scaling by $\sqrt{d_k}$ keeps raw dot-product scores from growing too large as the dimension grows; the softmax is applied row by row, each row answering one token's question about how much attention to give the others.

A Full Worked Attention Output

One query, two candidate tokens, single head, $d_k = 1$ to keep the arithmetic clean. Let

Raw scores are the dot products:

With $d_k = 1$ the scaling factor is $\sqrt{1} = 1$, so the scaled scores stay $4$ and $2$. Apply softmax:

The output is the weighted combination

The output lands closer to $v_1$ because the query matched $k_1$ more strongly. Real transformers do this in higher dimensions across many tokens at once, but the arithmetic is the same.

Self-Check at Each Step

• After scoring: are the scores dot products of this query with each key? Forgetting the scale factor $\sqrt{d_k}$ is a frequent slip.
• After softmax: do the weights sum to $1$? If not, the row-wise softmax was applied wrong.
• After the mix: does the output sit between the values, leaning toward the best-matched one? If it lands outside that range, the weighting is off.

What Else Lives in the Block

A standard block also has multi-head attention (several relationship types at once), a position-wise feedforward network, residual connections, and layer normalization. The original sequence-to-sequence design had an encoder stack (self-attention over the input) and a decoder stack (masked self-attention so a position cannot look ahead, plus cross-attention where queries come from the decoder and keys/values from the encoder). Many modern language models use only the decoder side, specialized for next-token prediction. Because attention alone is permutation-equivariant, transformers must add positional information so order-sensitive tasks work.

Where the Procedure Goes Wrong

• Thinking attention is the whole transformer. It is central, but feedforward layers, residual paths, normalization, and positional information all matter.
• Mixing up self-attention and cross-attention. In self-attention $Q$, $K$, $V$ share one sequence; in cross-attention they do not.
• Forgetting masking. Decoder-only models need causal masking so a token cannot attend to future tokens.
• Treating attention weights as a full explanation. Behavior also depends on value vectors, later layers, and nonlinear transforms.

Transformers fit language modeling, translation, summarization, code generation, speech, and many vision tasks — anywhere relationships across a sequence matter more than local patterns. Compute one small attention output by hand, follow the five steps once, and the leap from "I know the formula" to "I understand what the architecture is doing" usually happens right there.