Transformer Architecture Explained for Non-Researchers

In June 2017, eight researchers from Google Brain and Google Research published a paper with a title that reads like a dare: "Attention Is All You Need." The authors, including Ashish Vaswani, Noam Shazeer, and Illia Polosukhin, were making a claim radical even by the standards of a field accustomed to rapid revision. Eliminate recurrence entirely. Eliminate convolution. Use only attention mechanisms to process sequences.

This was not an incremental improvement on RNNs. It was a different answer to the same question about how to represent sequential information. The central architectural bet: if you let every position in a sequence attend to every other position in a single operation, you dissolve both the parallelization bottleneck and the long-range dependency problem simultaneously. Parallelization becomes possible because no step is waiting on a previous one. Long-range dependencies become trivial because any two tokens are one attention operation apart, not a chain of fifty sequential steps.

Worth pausing on: the paper was solving machine translation. Input text in one language, output text in another. The authors were not announcing a general-purpose AI substrate. That interpretation came later, from other researchers building on this foundation, and it took years to fully materialize. The gap between what a paper claims and what it eventually enables is something worth keeping in mind as you read through what follows.

Language is sequential, and it is relational. The sentence "The animal didn't cross the street because it was too tired" means something specific, and that meaning hinges on a relationship between "it" and "animal" spanning several words. The dominant approach before 2017 was the recurrent neural network, which processed text one word at a time, left to right, carrying a running summary forward into each new step. Elegant in theory. In practice, it produced two compounding problems that were not tunable because they were structural.

First, sequential processing cannot be parallelized. Each step waits for the previous one, and at modern training scales, that is not a bottleneck you can budget around. Second, the vanishing gradient problem: as information travels backward through many sequential steps during training, the signal degrades exponentially. The connection between "it" and "animal" blurs and dissolves across the intervening tokens. More data did not fix this. Hyperparameter tuning did not fix this. The 2017 paper did, by refusing to process sequentially at all.

How Raw Text Becomes Numbers the Model Can Work With

Before any attention can happen, language has to become something a mathematical operation can process. This part is not glamorous, and most explainers skip it, but skipping it means the rest of the architecture floats without a foundation.

Tokenization splits text into discrete units called tokens, roughly words or word-pieces depending on vocabulary design, so the model works with a finite, enumerable set of inputs. Each token is then mapped to an embedding, a vector of numbers positioning the token in a high-dimensional space where semantically similar words end up near each other. "Dog" and "puppy" sit closer together in that space than "dog" and "carburetor." The embedding is a learned representation; the model adjusts it throughout training.

Here is where things get quite strange, and where I see people get tripped up most often. Embeddings encode meaning, but they carry no information about where in a sentence a word appears. Self-attention, by design, is order-agnostic. Shuffle the tokens before feeding them in and the attention mechanism has no inherent way to know anything changed. Order is not implicit; it has to be injected separately. But what if that positional information were lost entirely — what would the model actually produce?

This is the positional encoding problem, and the original paper's solution was to add sinusoidal positional encodings to each token's embedding before it entered the model, a unique fixed mathematical pattern for each position giving the model a sense of sequence. Later architectures moved toward learnable positional embeddings. More recent approaches, including Rotary Position Embedding (RoPE), encode relative position directly into the attention computation itself, which generalizes better to sequences longer than those seen during training. The field is still actively arguing about which approach is best, which tells you the problem is not fully solved.

Each token enters the attention mechanism carrying both its meaning and its position. That combination is what the model actually works with.

How Self-Attention Lets Every Word Look at Every Other Word at Once

Each token generates three vectors from three learned projections: a Query, a Key, and a Value. The Query represents what this token is looking for. The Key represents what this token offers to others looking at it. The Value is the actual content it contributes when another token selects it. The naming is a bit unfortunate, because it implies more intentionality than is actually there; these are just learned linear transformations, not anything resembling intent.

To compute an attention score between two tokens, you take the dot product of one token's Query and the other's Key, scale it by the square root of the key dimension, then pass all those scores through a softmax that converts them into a probability distribution. The scaling matters: without it, large dot products push the softmax into near-zero gradient territory, which stalls learning in a way that's very hard to diagnose if you don't know to look for it. That distribution weights the Values. High attention weight means one token is strongly shaping another's representation.

Return to the "it was too tired" sentence. Self-attention computes a high relevance score between "it" and "animal" directly, regardless of the intervening tokens. No sequential chain, no degraded signal. One operation across the whole sequence, all at once.

What the model learns to do through all of this is next-token prediction: given all prior context, what token comes next? Self-attention is how "all prior context" gets computed efficiently and in parallel. That parallel computation is precisely what the RNN architecture never delivered, and it is why the 2017 paper mattered so much so quickly.

Why One Attention Pass Isn't Enough and What Multiple Heads Add

A single attention head produces one particular view of relationships in a sequence, one set of relevance scores governed by one learned set of Query, Key, and Value projections. Any sentence is simultaneously encoding syntactic structure, coreference, proximity, semantic similarity. A single head is forced to collapse all of that into one perspective, which is a real constraint, not a theoretical one. That raises an important question: what does the model actually lose when it can only ask one question at a time?

Multi-head attention runs several attention heads in parallel, each with its own learned projections, then concatenates and projects the results back into a single representation. One head specializes in subject-verb agreement. Another handles coreference. Another attends to local word proximity. Critically, the model learns which specializations are useful during training; nobody hand-codes these distinctions, and in practice the specializations that emerge are often not the clean linguistic categories you expect.

The distinction worth holding: multi-head attention is not a different mechanism from single-head attention. It is the same mechanism asked several questions simultaneously. That precision matters when you are trying to reason about what a model is actually doing, rather than what someone's marketing copy says it's doing.

What the Feed-Forward Layers, Residual Connections, and Layer Norm Actually Do

Attention decides which information to gather from other tokens. It does not transform what was gathered. That work falls to the feed-forward network following each attention operation: two linear transformations with a nonlinearity between them, applied independently to each token's position. If attention is how tokens consult each other, the feed-forward layer is where the model processes the results of that consultation.

Two additional components make it possible to stack many such layers without the training process destabilizing, and both are borrowed from prior work rather than invented here.

Residual connections, adapted from the ResNet computer vision architecture developed in 2015, add the input to each sub-layer back to that sub-layer's output. The effect is that gradients have a direct backward path through the network even as it gets very deep. Without them, training a model with many stacked layers produces gradients that vanish or explode long before reaching the early layers. This is a version of the same structural problem RNNs faced, and residual connections are why transformers escape it.

Layer normalization stabilizes the distribution of activations across each layer, preventing training from becoming chaotic as representations shift across many operations. Modern architectures have converged on applying normalization before each sub-layer rather than after, as the original paper specified, because earlier application proves more stable in practice. The fact that practitioners changed this detail quietly and relatively quickly suggests the original paper was a starting point, not a finished specification.

Depth is where the transformer's expressive capacity actually comes from. These three components are what make depth tractable rather than theoretically desirable but practically disastrous.

How the Encoder-Decoder Structure Connects Input Understanding to Output Generation

The 2017 transformer was designed for translation, and that task has a natural shape: a full input sequence in one language, an output sequence in another. The encoder-decoder structure maps directly onto this shape, which is either elegant or obvious depending on how much translation systems you've worked with.

The encoder is a stack of multi-head attention and feed-forward layers that reads the entire input sequence. Because every token can attend to every other token in the input, this is bidirectional attention; the representation of each word is shaped by everything around it simultaneously. The output is a rich contextual representation of the source sequence.

The decoder generates the output sequence one token at a time and uses two kinds of attention. Masked self-attention lets the decoder attend to positions it has already generated, but not to future ones; during training, this prevents the model from simply copying the correct answer from tokens it is supposed to be predicting. Cross-attention lets the decoder attend to the encoder's output, which is how the decoder reads the source material while constructing the target.

At the end of the decoder, a linear layer followed by a softmax converts the final representation into a probability distribution over the vocabulary. The model selects the next token from that distribution.

The encoder-decoder design was a reasonable choice for translation. It was not the only possible choice, and the researchers who came after 2017 noticed that different tasks called for different pieces of this structure, often just one piece.

How BERT, GPT, and Their Descendants Each Use Only Part of That Original Design

The encoder and decoder are separable, and separating them turned out to be productive in ways that were not obvious beforehand.

BERT-style models use only the encoder stack. Bidirectional attention means every token sees all other tokens at once, which makes these models well-suited for tasks requiring understanding rather than generation: classification, named-entity recognition, question answering where the answer exists in a source document. The model reads everything before it decides anything.

GPT-style models use only the decoder stack, with masked left-to-right attention. The entire training objective is predicting the next token. GPT-3 demonstrated something that surprised researchers who already expected scaling to matter: pushing a decoder-only model to very large parameter counts produced qualitative capability jumps, not just incremental improvements. Capabilities that simply did not exist at smaller scales emerged at larger ones. The mechanism behind this is still not fully understood, which is an uncomfortable thing to say about systems now embedded in production infrastructure worldwide.

Encoder-decoder models like T5 retained the full original structure, suited for tasks where a complete input maps explicitly to a different complete output: translation, summarization.

The evolution has continued. GPT-4 extended the architecture to multimodal inputs; GPT-4o unified text, vision, and audio in a single model. The o1 and DeepSeek R1 families introduced internal chain-of-thought reasoning as an additional layer, allowing models to reason through steps before producing output. The core attention mechanism persists across all of these variants. What changes is the surrounding structure and the training regime.

For anyone building on top of these systems, which half of the encoder-decoder design a model uses tells you something predictive about what it is optimized for. This is not a marketing distinction.

The Efficiency Problem That Scale Exposed and How Modern Architectures Respond

Self-attention's power derives from its pairwise nature: every token attends to every other token. That relationship is what makes it so expressive, and it is also why computational cost scales quadratically with sequence length. Double the context window and the attention computation does not double; it quadruples. At very long contexts, this is a hard infrastructure constraint, not a theoretical concern that larger budgets resolve.

Three engineering responses have become standard, and they address the problem from different angles.

Sparse attention limits computation to a strategically chosen subset of token pairs rather than all pairs. The majority of useful signal is preserved while computational cost drops considerably; the difficult question is which pairs to drop, and different architectures make different bets here.

Efficient attention algorithms, FlashAttention being the prominent example, do not reduce the number of operations so much as restructure how those operations use hardware memory. Memory reads and writes, not raw arithmetic, are often the actual bottleneck in attention computation, and FlashAttention addresses that at the implementation level. This is a case where systems-level engineering produced gains that model architecture changes alone could not.

Mixture of Experts approaches replace a single dense feed-forward block with many specialized sub-networks and a learned router that sends each token to only a small number of them. This decouples total model capacity from per-token compute cost. Gemma 4's MoE model carries between 26 and 27 billion total parameters but activates only around 4 billion per token, delivering larger-model reasoning quality at smaller-model inference cost. More than 60 percent of frontier models released in 2025 use MoE architectures, which tells you the community has largely converged on this approach even while debating the details.

These are optimizations within the transformer paradigm. The self-attention mechanism at the core remains intact.

The Limitations That Engineering Optimizations Cannot Fix

Not all transformer limitations are resource problems. Conflating the two categories produces bad expectations and, downstream, bad product decisions.

Hallucination is the instructive case. Mathematical analysis of how transformers generate outputs shows that producing confident-sounding wrong answers is not primarily a data quality failure or a fine-tuning deficiency. It is a structural property. The model is sampling from a probability distribution, and that distribution is highly confident about an incorrect token. One might argue that better training data would eventually close this gap; the problem with that argument is that the issue is not what the model has seen but how the model generates, token by token, from a distribution that has no mechanism to verify factual accuracy. Detection and grounding strategies reduce hallucination frequency. They cannot eliminate the structural possibility of it, and representing them as capable of doing so is a form of architectural illiteracy that has caused real production failures.

Compositionality gaps are a separate concern. Transformers trained on large corpora acquire impressive surface-level reasoning capabilities, but research has documented consistent failures on tasks requiring multiple reasoning steps chained in sequence, specifically in function composition problems of relatively modest complexity. This is not an edge case that better models will quietly patch. It points to something about how attention-based sequence modeling represents multi-step logical dependencies.

Positional bias also persists. Models trained on sequences of a given length behave inconsistently on longer sequences, and this does not reliably resolve with better positional encoding schemes, despite significant effort from multiple research groups.

These are the real motivations behind active research into post-transformer approaches: state-space models, linear attention, revamped recurrent architectures aimed at sub-quadratic complexity and better compositional reasoning. None of these has displaced the transformer at scale as of 2025. The research trajectory is real, though, and it is driven by problems that inference optimization cannot reach.

Where the Transformer Architecture Has Reached Beyond Language

Self-attention does not require language. It requires sequences and meaningful relationships between elements in those sequences. This is obvious once you see it happen, but it was not obvious in 2017, and the breadth of the generalization has repeatedly exceeded what researchers anticipated.

Vision Transformers treat an image as a sequence of fixed-size patches. Standard transformer attention runs across those patches and has matched or surpassed convolutional networks on image classification benchmarks. The architecture transferred almost directly, because the underlying problem structure turned out to be the same; spatial relationships in images and sequential relationships in text are, at some level of abstraction, the same problem.

Protein folding is a more consequential case, and the one that shifted how I thought about where this architecture's ceiling is. Amino acid sequences behave, in a meaningful sense, like a biological language, carrying evolutionary patterns and structural information in the ordering of their elements. AlphaFold's Evoformer architecture uses attention-based deep networks to extract these patterns and predict three-dimensional protein structure from sequence alone. This work was recognized with the 2024 Nobel Prize in Chemistry. In drug discovery, the predicted structures accelerate identification of how therapeutic molecules bind to their targets, compressing a research phase that previously took years into something closer to weeks.

The pattern is consistent: any domain with structured sequences and meaningful long-range dependencies is a plausible candidate for transformer-style attention. The architecture traveled farther from its original context than its authors anticipated, and the journey is probably not over.

On the access side, the Hugging Face ecosystem and Meta's Llama family, released from 2023 onward, gave developers substantive access to transformer models without requiring dependence on closed corporate APIs. The architecture is now a broadly available building block, not a capability gatekept by a handful of companies.

What the Component-by-Component Picture Adds Up to for People Building With Transformers Today

There is a specific kind of leverage you get from understanding architecture rather than just behavior. When a model surprises you, with a capability or a failure, you have somewhere to look.

The quadratic scaling of attention is not an abstraction; it is the reason context window limits matter more than they appear and why long-document tasks behave differently from short ones in ways that are not explained by the benchmark numbers. Hallucination is not randomness; it is a structural feature of probability distribution sampling, which means detection and grounding strategies are more tractable than elimination strategies, and anyone selling you elimination is selling you something the architecture cannot deliver. The difference between an encoder-only and decoder-only model predicts which tasks each will handle well and which will frustrate you in ways that no amount of prompting resolves.

For teams evaluating or deploying language model infrastructure, this mechanical understanding shapes vendor selection in concrete ways. Tools like Weights and Biases for experiment tracking, or inference and fine-tuning platforms like Replicate or Modal, are better evaluated when you understand what the model is doing underneath. Knowing that a given frontier model uses an MoE architecture changes your inference cost assumptions significantly, before you have committed to a contract or an architecture decision.

The 2017 paper solved a specific, concrete problem: how to process sequences without sequential processing. Every component it introduced was a direct answer to either that problem or a problem that solution created. Self-attention gave every token visibility into every other. Multiple heads gave the model parallel perspectives on the same sequence. Feed-forward layers processed the gathered context. Residual connections and layer normalization made depth trainable. The encoder processed input; the decoder generated output.

What followed, BERT, GPT, ViT, AlphaFold, the MoE frontier models, all of it sits on that foundation. Not because no one has tried to replace it, but because the original architectural bet was sound in ways that only became clear at scale. The benchmarks, the product announcements, the capability claims: they become considerably easier to evaluate once you understand why each piece exists and what it was designed to solve.