Transformer Architecture for Engineers — Attention, Layers & Scale (2026)

Every skill in modern GenAI engineering — RAG, agents, fine-tuning, evaluation — rests on the same substrate: the Transformer architecture. Understanding it is not an academic exercise. It is the prerequisite for reasoning clearly about why your system behaves the way it does, why certain prompts work, and why certain architectures fail at scale.

Who this is for:

• Software engineers moving into GenAI: You have shipped production software. You need to understand the mechanics of the architecture, not just how to call the API.
• Junior GenAI engineers: You have built basic LLM apps. You want the architectural depth to explain your decisions in design reviews and interviews.
• Senior engineers preparing for interviews: You need to explain self-attention, QKV matrices, and encoder vs. decoder trade-offs under pressure, with precision.

The Transformer was introduced in the 2017 paper “Attention Is All You Need” by Vaswani et al. at Google. Before it, sequence modeling relied on recurrent neural networks (RNNs) and LSTMs, which processed text token by token in sequence. This made training slow and made it hard for models to relate tokens that appeared far apart in a sentence.

The Transformer replaced recurrence with attention. Instead of processing tokens one at a time, it processes every token in parallel and lets each token directly attend to every other token in the sequence. This was a decisive shift — it enabled the scale-up in training compute that produced GPT-3, GPT-4, Claude, Gemini, and Llama.

As a GenAI engineer, you do not need to implement a Transformer from scratch. But you do need the architecture intuition to:

• Debug context problems. When a model “forgets” something in a long context, that is attention dilution — a consequence of how attention scores spread across thousands of tokens.
• Choose the right model type. BERT-class models and GPT-class models are built on the same foundation but make opposite design choices. Picking the wrong one for your task is a common engineering mistake.
• Reason about cost and latency. The quadratic cost of attention with respect to sequence length is not an implementation detail — it is the central constraint in context window design.
• Understand why fine-tuning works. The layers of a Transformer store different things. Knowing which layers encode what helps you understand why adapter-based fine-tuning is efficient.

You already use Transformers every day when you call the OpenAI or Anthropic API. Understanding what happens inside the model turns you from a user of the API into an engineer who can reason about it.

The attention mechanism is the single most important concept in the Transformer architecture. Everything else — multiple heads, layers, encoder vs. decoder — is built on top of it.

Consider the sentence: “The server crashed because it ran out of memory.”

The word “it” is ambiguous in isolation. To resolve “it” correctly, the model needs to connect it to “the server,” not to “memory.” A sequential model (RNN) processes “it” without directly comparing it to “the server” — it relies on what it has accumulated in a hidden state by that point, which degrades over long distances. Attention solves this by computing a direct relationship between “it” and every other word, including “the server.”

For each token in a sequence, the attention mechanism creates three vectors:

• Query (Q): “What information am I looking for?” The query represents what the current token needs to understand itself.
• Key (K): “What information do I contain?” The key represents what a token can offer to other tokens.
• Value (V): “What information should I contribute if selected?” The value is the actual content that gets passed forward when attention is high.

These vectors are computed by multiplying the token’s embedding by three learned weight matrices: W_Q, W_K, and W_V.

Attention scores are computed in four steps:

1. Dot product: Multiply Q by the transpose of K for all token pairs. High dot product = the query vector aligns with the key vector = high relevance.
2. Scale: Divide by the square root of the key dimension (√d_k). Without this, dot products become large in high dimensions and the softmax saturates, making gradients vanish during training.
3. Softmax: Convert raw scores to probabilities. Each token now has a probability distribution over all other tokens — how much it should “attend to” each one.
4. Weighted sum: Multiply the softmax output by V. The result is a new representation for each token that is a weighted combination of the values of all tokens, where the weights reflect relevance.

In formula form:

Attention(Q, K, V) = softmax(QK^T / √d_k) × V

This is the entire attention mechanism. Three learned matrices, four operations. The output for each token is a context-aware representation that has “looked at” every other token and weighted their contributions.

If you are using a model with a very long context window (100K+ tokens), the QK^T matrix becomes enormous — it is N×N where N is the number of tokens. At 128K tokens, that is 128,000 × 128,000 = ~16 billion attention score values computed per head per layer. This is why context window length has such a large impact on inference cost and latency.

Each Transformer block stacks multi-head self-attention, residual connections, layer normalization, and a feed-forward network — repeated dozens to hundreds of times in modern LLMs.

Multi-Head Self-Attention is the layer described in the previous section. It lets every token gather contextual information from the rest of the sequence. Without this layer, the model would process each token as if it existed in isolation.

Residual connections (the “Add” steps) add the input of a sub-layer directly to its output. This solves the vanishing gradient problem in deep networks and lets information pass through without distortion. Without residual connections, a 96-layer model would be nearly untrainable.

Layer Normalization normalizes the distribution of activations across the embedding dimension. This stabilizes training and makes the model less sensitive to learning rate choices. Pre-norm (applied before the sub-layer) is the modern standard; it is used in GPT-style models.

Feed-Forward Networks are applied independently to each token after attention. Despite being “per-token,” they are not trivial — research suggests this is where factual knowledge is stored in the model’s weights. The FFN expands the dimension (typically 4x), applies a nonlinear activation (GELU in modern models), and projects back down.

The full Transformer model stacks this block repeatedly. GPT-3 has 96 layers. Estimated GPT-4 architecture has approximately 120 layers in each expert. Each pass through the stack lets the model build progressively richer representations — early layers capture syntax and local patterns; later layers encode semantic relationships and world knowledge.

Single-head attention computes one set of attention weights per layer. This is powerful, but limited — a single set of weights can only capture one “type” of relationship between tokens at a time.

Multi-head attention runs the attention mechanism h times in parallel, with different learned weight matrices for each head. The outputs are concatenated and linearly projected back to the original dimension.

Each head develops its own specialization. Research using attention visualization tools (like BertViz) has found that different heads in trained models tend to capture:

• Syntactic dependencies: Which noun does this verb agree with? Which adjective modifies which noun?
• Coreference resolution: “it” refers to “the server,” not “the memory.”
• Positional patterns: Some heads attend primarily to adjacent tokens or tokens at fixed offsets.
• Semantic similarity: Heads that activate strongly when tokens share domain-specific meaning.

GPT-3 uses 96 attention heads per layer. This breadth lets the model simultaneously track dozens of different types of relationships when generating each token.

With d_model = 12,288 (estimated for GPT-4) and 96 heads, each head operates on a 128-dimensional subspace (12,288 / 96). The full model dimension is preserved through the concatenation-and-projection step. This means multi-head attention does not cost more than single-head attention of the same total dimension — the compute is the same, but the representational capacity is much higher.

When a model fails to capture a specific kind of relationship — say, a coreference across a very long span — it is often because the attention heads responsible for that pattern are “overwhelmed” by shorter-range dependencies in the same layer. This is one reason why structured prompting (keeping relevant context close together) improves model performance. You are working with the grain of how attention heads organize information, not against it.

Self-attention is permutation-invariant by design. If you shuffle all the tokens in your prompt, the attention scores between any two tokens remain identical — because they depend only on the Q and K vectors, which are derived from the tokens themselves and not their positions. “The cat sat on the mat” and “mat the on sat cat the” would produce the same attention patterns without positional encoding.

Positional encoding adds position-dependent information to each token’s embedding before the first attention layer. Three major approaches:

The original 2017 paper used deterministic sinusoidal functions of different frequencies to encode each position. Position p and dimension i get the encoding:

PE(p, 2i)   = sin(p / 10000^(2i/d_model))
PE(p, 2i+1) = cos(p / 10000^(2i/d_model))

The key properties: each position gets a unique vector, nearby positions get similar vectors, and the encoding generalizes to sequences longer than any seen in training. The limitation: it was designed for the original encoder-decoder architecture and does not transfer cleanly to the modern relative attention patterns used in decoder-only models.

GPT-2 and early GPT-3 used a learned position embedding table — a matrix of shape (max_sequence_length, d_model) where each row is a trainable vector for position p. This is simple and effective. The downside is that it hard-codes a maximum sequence length at training time. A model trained with 2,048-position embeddings cannot, in principle, generalize to position 2,049.

RoPE, introduced by Su et al. in 2021 and adopted by Llama, Mistral, and many other modern open-weight models, encodes position by rotating the Q and K vectors in the complex plane. The relative angle between two vectors encodes their positional distance rather than absolute position.

Key properties of RoPE:

• Relative, not absolute: The model learns “how far apart are these tokens” rather than “which absolute position is this token.” This is more transferable to longer contexts.
• Extrapolation: With techniques like YaRN (Yet another RoPE extensioN), models can extend their effective context window significantly beyond their training length.
• No additional parameters: The rotation is deterministic given the position index. No learned embedding table required.

ALiBi, used in BLOOM and some MPT models, does not modify embeddings at all. Instead, it adds a linear penalty to attention scores based on distance: tokens far apart receive lower attention scores, regardless of their content relationship. The penalty slope is a fixed hyperparameter. This enables excellent length generalization — ALiBi models trained on 2K tokens can often generalize to 8K+ with acceptable performance degradation.

For engineers using pre-trained models: the positional encoding scheme determines how well a model extrapolates beyond its training context length. If you are working with very long documents and a model trained with absolute learned embeddings, you will see quality degradation when inputs approach or exceed the training length. Models using RoPE or ALiBi degrade more gracefully.

The original Transformer had two components: an encoder and a decoder. Modern LLMs have largely split into two distinct lineages, each dropping one of these components.

Encoder-only models use bidirectional attention — every token can attend to every other token in both directions. The [CLS] token’s representation aggregates information from the full input, making it ideal for classification.

Training objective: Masked Language Modeling (MLM). Randomly mask 15% of input tokens; train the model to predict the masked tokens from context. This forces the model to develop rich contextual representations.

Best use cases:

• Semantic search and dense retrieval (producing embeddings)
• Text classification and sentiment analysis
• Named entity recognition (NER)
• Natural language inference

Limitation: encoder-only models do not generate text well. They are not trained to predict the next token in a left-to-right sequence — they cannot produce coherent continuations.

Decoder-only models use causal (autoregressive) attention — each token can only attend to tokens that appeared before it in the sequence. This is enforced by a triangular mask applied to the attention matrix before softmax.

Training objective: Next-token prediction (CLM). Given tokens 1 through N-1, predict token N. This is maximally simple and scales well.

Why decoder-only won for LLMs: generation is the core use case. GPT-3’s release in 2020 demonstrated that sufficiently large decoder-only models can perform classification, translation, summarization, and question answering through prompting alone — without task-specific fine-tuning. The generation capability is not a limitation; it is the feature.

The original Transformer’s encoder-decoder architecture introduces a third type of attention: cross-attention. The decoder computes Q from its own representations, but K and V from the encoder’s output. This lets the decoder “look at” the encoded input at every step of generation.

For translation: the encoder processes the French sentence bidirectionally; the decoder generates the English translation one token at a time, attending to the French encodings to determine what to say next.

The core Transformer architecture has remained largely stable since 2017. What changed to produce GPT-4, Claude 3.5 Sonnet, and Gemini 2.0 is not the fundamental design — it is what surrounds it.

The 2020 Kaplan et al. paper (from OpenAI) established empirical scaling laws: model performance on language tasks improves predictably with increases in model parameters, training data volume, and training compute — following smooth power laws. This provided a roadmap: to double model quality on benchmarks, roughly double the parameters and training data together.

The 2022 Chinchilla paper from DeepMind refined this: earlier large models were significantly undertrained relative to their parameter count. Chinchilla showed that for a given compute budget, you get better results by training a smaller model on more data than a larger model on less data. This shifted the field toward data-efficient training.

For engineers: scaling laws explain why the biggest model is not always the best model. A 70B model trained on 2 trillion tokens often outperforms a 175B model trained on 300 billion tokens on many benchmarks.

During autoregressive generation, the model computes K and V matrices for every token in the context at every generation step. Without optimization, generating a 1,000-token response would require 1,000 full forward passes, each recomputing K and V for all previous tokens from scratch.

KV caching solves this: K and V matrices for the tokens already processed are stored in memory and reused at subsequent steps. Only the new token’s K and V need to be computed at each step. This is why generation speed scales primarily with the number of tokens being generated, not the total context length.

Engineers working with self-hosted models (vLLM, TensorRT-LLM, Ollama) tune KV cache memory allocation carefully. The KV cache for a 128K-token context with a 70B model at float16 can consume tens of gigabytes of GPU VRAM — more than the model weights themselves. This is a central constraint in production serving infrastructure.

GPT-4 is widely believed to use a Mixture of Experts architecture. Instead of activating all model parameters for every token, MoE routes each token to a subset of “expert” feed-forward networks (typically 2 out of 8 or 2 out of 64). This allows a model with hundreds of billions of parameters to have the quality of a dense model of that size while only activating a fraction of parameters per token.

The tradeoff: total parameter storage increases (all experts must fit in memory), but per-token compute decreases. Mistral’s Mixtral 8x7B demonstrates this openly — 46B total parameters, but only ~12B activated per token.

For engineers: MoE explains why model size in parameters and inference cost are not proportional. A 70B MoE model can be faster at inference than a 13B dense model with the right serving setup.

Pre-trained Transformer models (often called base models) are trained only on next-token prediction. They will complete prompts, but they will not follow instructions, refuse harmful requests, or maintain a consistent assistant persona. The post-training stack transforms them:

1. Supervised Fine-Tuning (SFT): Train on high-quality instruction-response pairs. The model learns to follow the instruction format.
2. Reward Model Training: Human annotators rank model outputs. A separate reward model is trained to predict human preference scores.
3. RLHF (Reinforcement Learning from Human Feedback): The SFT model is optimized using the reward model’s signal via PPO or GRPO. The model learns to generate outputs that humans prefer.
4. Constitutional AI / DPO: Anthropic’s Constitutional AI and Direct Preference Optimization are newer approaches that reduce reliance on expensive RLHF, improving efficiency and stability.

For engineers: the model you call via the API is not the base model. It has been significantly shaped by RLHF to be helpful, harmless, and honest. This shapes its behavior in ways that matter for prompt engineering — the model has learned strong preferences for instruction-following formats, specific reasoning styles, and certain output structures.

Transformer architecture questions appear at every seniority level in GenAI engineering interviews. The depth expected scales with role level.

What interviewers are testing: Can you build intuition without hiding behind equations? Can you teach this concept?

Strong answer:

“When a model reads a sentence, it needs to understand what each word means in context. The word ‘bank’ means something different in ‘river bank’ versus ‘investment bank’ — and the model needs to figure that out from surrounding tokens.

Self-attention does this by letting every token ask: ‘Which other tokens in this sequence are most relevant to understanding me?’ Each token creates a query (what I’m looking for), a key (what I offer to others), and a value (what I contribute when selected). Relevance is computed as a dot product between a token’s query and every other token’s key. High-scoring tokens contribute more of their value to the current token’s updated representation.

The result: each token’s representation after attention is a weighted mix of information from the entire sequence, not just the token itself. ‘It’ in ‘The server crashed because it ran out of memory’ will, after attention, incorporate information from ‘the server’ — enabling correct coreference resolution.”

What interviewers are testing: Architecture decision-making, not just recall.

Strong answer:

“Encoder-only models like BERT use bidirectional attention — every token sees every other token. They produce rich contextual embeddings ideal for understanding tasks: classification, named entity recognition, semantic search. If I am building a retrieval system that needs to embed documents and queries into a shared vector space, I reach for an encoder-only model.

Decoder-only models like GPT, Claude, and Llama use causal attention — each token can only see tokens before it. They are optimized for generation: text completion, chat, code. All current frontier LLMs are decoder-only because generation is the dominant use case, and they have demonstrated strong understanding capabilities through prompting alone.

I would use an encoder-only model when I need high-quality dense embeddings for retrieval or when I need a discriminative classifier. I would use a decoder-only model for any generative task. For very long translation or structured summarization, an encoder-decoder model might still be the right choice.”

What interviewers are testing: Understanding of computational complexity and its engineering implications.

Strong answer:

“Self-attention requires computing a score for every pair of tokens in the sequence. With N tokens, that is N squared comparisons per head per layer. For a model with 96 heads and 120 layers processing a 128K-token context, the raw attention computation is 128,000 squared × 96 × 120 — roughly 240 trillion operations, though Flash Attention and memory-efficient implementations reduce this significantly in practice.

The engineering implications are real: doubling the context length roughly quadruples the memory required for attention scores and significantly increases computation time. This is why context windows are not free — Gemini 2.0 Pro’s 2M token window is impressive, but using it is expensive both in dollars and in latency.

For production system design, this shapes the RAG vs. long-context decision. For most corpora, targeted retrieval into a smaller context is cheaper, faster, and often more accurate than stuffing everything into a huge context window.”

What interviewers are testing: Understanding of inference optimization beyond just calling the API.

Strong answer:

“During autoregressive generation, the model needs the K and V matrices for every previously generated token to compute attention for the current token. Without caching, each generation step would recompute K and V for the entire context from scratch — that would make output token generation O(N²) per token.

KV caching solves this by storing K and V matrices after they are computed and reusing them for subsequent tokens. Only the new token’s K and V need to be computed at each step. This makes generation O(1) per token in terms of new compute, at the cost of memory proportional to context length.

The engineering implication: in a production serving environment, KV cache memory becomes the binding constraint, not model weight memory. A 70B model serving 128K-token contexts needs tens of gigabytes of KV cache per request. Systems like vLLM’s PagedAttention manage KV cache memory like virtual memory pages — sharing cache across requests with common prefixes to increase throughput. This is one of the most important optimizations in modern LLM serving infrastructure.”

Understanding the Transformer architecture unlocks everything else in GenAI engineering. With this foundation:

• LLM Fundamentals — Reinforce tokenization, temperature, and the end-to-end generation pipeline with practical engineering examples.
• Embeddings — Go deeper into how encoder-only Transformers produce the dense vectors that power semantic search, RAG retrieval, and clustering.
• Context Windows — Understand the practical engineering trade-offs of long-context models, including the lost-in-the-middle problem and cost implications.
• Fine-Tuning — Learn how post-training adapts a base Transformer for specific tasks — from full fine-tuning to LoRA and QLoRA.
• LLM Evaluation — Build the testing infrastructure that measures whether your model changes improve real-world quality, not just benchmark scores.

• LLM Fundamentals — Tokens, attention, and the generation pipeline
• Tokenization Guide — BPE, SentencePiece, and token counting
• Embeddings Explained — How encoder models produce dense vectors
• Context Windows — Engineering trade-offs of long-context models