How Transformers Work: The Design Behind Language Models, Explained with Code

ChatGPT, GPT-4, Claude, Llama and Google’s Bard look like very different products. Underneath, they share one design: the Transformer. It was first described in 2017 in a research paper titled “Attention Is All You Need”, originally to improve machine translation. Within a few years it had replaced almost every other approach to language, and it is now used for images, audio and code as well.

You do not need to train a model to benefit from understanding it. Several things that confuse people about language models, why they forget details in long documents, why cost grows so fast with prompt length, why prompt injection is so hard to stop, follow directly from how a Transformer works.

The problem it solved

Before the Transformer, the leading models for language read text one word at a time, in order, carrying a running summary forward. These were recurrent neural networks. They had two weaknesses.

They were slow to train. Each word had to wait for the previous one to be processed, so the work could not be spread across many processors at once.

They forgot. By the end of a long sentence, information from the beginning had been squeezed through many steps and was often lost. In “The keys that I left on the kitchen table next to the bills are missing”, connecting “are” back to “keys” is hard for a model that reads strictly left to right.

The Transformer removed the word-by-word chain. It looks at all the words at once and lets every word directly consult every other word. That mechanism is called attention.

Step 1: Text becomes tokens, and tokens become numbers

The model does not see words. A tokenizer first splits text into tokens, which are common words or pieces of words. “Unbelievable” might become “un”, “believ” and “able”. In English, one token is roughly four characters, or about three quarters of a word.

Each token is then mapped to a list of numbers called an embedding. In GPT-3, each token becomes a list of 12,288 numbers. During training, tokens with related meanings end up with similar lists.

Step 2: Add the position

Because the Transformer looks at all tokens at once, it has no built-in sense of order. “Dog bites man” and “man bites dog” contain the same tokens. To fix this, position information is added to each token’s embedding, so the model knows that a token is first, fifth or five hundredth. The original paper used a fixed pattern of sine and cosine waves. Many newer models learn position information during training instead.

Step 3: Attention, the core idea

For every token, attention answers one question: which other tokens should I pay attention to, and how much?

Take the sentence “The animal didn’t cross the street because it was too tired.” To understand “it”, the model must connect it to “animal”, not “street”. Change “tired” to “wide” and “it” now means the street. Attention is how the model makes that connection.

Each token produces three lists of numbers, by multiplying its embedding by three sets of learned weights:

• A query: what this token is looking for.
• A key: what this token offers to others.
• A value: the information this token passes on if it is chosen.

To decide how much “it” should attend to “animal”, the model compares the query of “it” with the key of “animal”. A strong match means high attention. The scores for all tokens are turned into weights that add up to 1, and the token’s new representation is the weighted mix of everyone’s values.

In the paper, the whole step is one formula:

Attention(Q, K, V) = softmax(Q × Kᵀ / √d) × V

In plain words: compare every query with every key, scale the scores down so they stay stable, turn them into percentages, and use those percentages to blend the values.

Here is the same thing as working Python code:

import numpy as np

def softmax(x):
    x = x - x.max(axis=-1, keepdims=True)    # for numerical stability
    e = np.exp(x)
    return e / e.sum(axis=-1, keepdims=True)

def attention(Q, K, V, causal=False):
    d = Q.shape[-1]
    scores = Q @ K.T / np.sqrt(d)           # how well each query matches each key
    if causal:
        n = scores.shape[0]
        future = np.triu(np.ones((n, n)), k=1).astype(bool)
        scores = np.where(future, -1e9, scores)   # a token may not look ahead
    weights = softmax(scores)               # each row adds up to 1
    return weights @ V, weights

rng = np.random.default_rng(0)
tokens = ["the", "cat", "sat", "down"]
d_model = 8

X = rng.normal(size=(len(tokens), d_model))       # stand-in token embeddings
W_q, W_k, W_v = (rng.normal(size=(d_model, d_model)) for _ in range(3))

output, weights = attention(X @ W_q, X @ W_k, X @ W_v, causal=True)

for token, row in zip(tokens, weights):
    print(token.ljust(5), np.round(row, 2))

Run it and you get a 4 by 4 table of weights. Each row shows how much one token attends to each token. Because causal=True, every row has zeros to the right of its own position: “sat” can look at “the”, “cat” and itself, but not at “down”. That mask is what lets a model like GPT generate text one token at a time without seeing the answer in advance. In a trained model, the weight matrices are learned from data instead of being random, and the patterns in this table become meaningful.

Step 4: Many heads, many layers

One attention calculation captures one kind of relationship. The Transformer runs several in parallel, called heads. One head might learn to link pronouns to nouns, another to link verbs to their objects, another to track position. The original base model used 8 heads. GPT-3 uses 96.

After attention, each token passes through a small feed-forward network, the part where much of the model’s stored knowledge appears to live. Two technical details keep training stable: a shortcut that adds each layer’s input to its output, and a normalization step.

Attention plus the feed-forward network makes one layer. Layers are stacked. The original model had 6 layers in each half. GPT-3 has 96 layers and 175 billion learned numbers, called parameters.

Step 5: Predict the next token

At the top of the stack, the model turns the final representation into a probability for every token in its vocabulary. For a model like GPT, training is simple to describe: show it huge amounts of text, have it predict the next token at every position, and adjust the parameters to make the correct token more likely. Repeat billions of times.

Generating text is the same step in a loop: predict a probability for every possible next token, pick one, add it to the input, and predict again. The temperature setting controls how that pick is made. Low temperature picks the most likely tokens. High temperature allows less likely ones.

Three families of Transformers

• Encoder models, such as Google’s BERT, read the whole text in both directions at once. They are strong at understanding tasks: search, classification, finding names in text.
• Decoder models, such as the GPT family, read left to right with the causal mask and generate text. Most chatbots are decoder models.
• Encoder-decoder models, such as the original Transformer and Google’s T5, read an input with one half and write an output with the other. Translation and summarization fit this shape.

What this means if you build on language models

Long prompts cost more than you expect. Every token attends to every other token, so the attention work grows with the square of the length. Twice the text is roughly four times the attention work. This is why context windows have hard limits, why longer windows are priced higher, and why trimming a prompt is often the cheapest optimization available.

Position in the prompt matters. A research paper released in July 2023 found that models use information at the start and end of a long context much better than information buried in the middle. Put the instructions and the most important material at the beginning or the end, not in the middle of a long document.

The model predicts likely text, not true text. Nothing in the design checks facts. The output is the most probable continuation of the input, which is why fluent invented answers happen and why grounding answers in real documents works so well.

There is no separate channel for instructions. This is the point security teams most need to understand. Your system prompt, the user’s message and the document being summarized become one sequence of tokens, and every token can attend to every other one. There is no wall in the architecture between instructions and data. That is the root cause of prompt injection, and it is why defenses have to live outside the model: validating output, limiting what the model can do, and keeping a person in the loop for actions that matter.

The model itself is a file. The weights are a large set of numbers saved to disk. Protect them like source code, and load them only from sources you trust and in safe formats.

Where to go next

If you read one primary source, read the original paper, “Attention Is All You Need” (2017). It is short, and with the steps above in mind most of it is readable. Then run the code in this article, change the tokens, turn the causal mask off and on, and watch the table change. Ten minutes with that table teaches more about attention than any diagram.