[Training a 500B LLM from Scratch] 4/25: What Is Inside a Large Language Model?

The purchase sheet had GPUs, the schedule had training time, and the budget had money. When it came to drawing the model, the page still held just one number: 500B.

How should those 500 billion parameters connect? I set the purchase sheet aside and wrote: “Rain outside. I picked up my…” Then I followed that sentence through the network. Until I could explain how “umbrella” emerged, more machines would still leave a black box inside.

What Does a Sentence Become When It Enters the Machine?

Text cannot take part in GPU multiplications directly. At the entrance, the sentence is split into small pieces called tokens. Each piece gets an ID, which looks up a row of numbers in a table. Every text position now has a numerical representation for the rest of the network to process.

Before following those numbers, I name the two main steps in each layer. Blue A is attention: it retrieves relevant information from earlier text for the current position. Gold B is the feed-forward network: it further processes the state that the current position now carries. We will follow the numbers first, then open A and B separately.

I marked the final position and followed it throughout. In the English diagram, it is “my.” The initial lookup gives it [0.2, -0.4, 0.7]. After blue A, information from “Rain outside” participates in the calculation, changing the numbers to [0.5, 0.1, 0.6]. Gold B then produces [0.8, -0.2, 0.9]. Those three numbers enter the next layer unchanged and continue through the network.

Each cell is a number; the three cells together are a vector. Only three components are shown so the changes remain visible. Real models commonly use thousands of numbers for one position. The text at the end has not changed, but the numbers it carries are changing. As earlier information arrives, its representation develops meaning within this sentence.

That separated two things for me. Weights retained after training determine how to calculate. The changing numbers in front of us are states produced while processing this sentence. During generation, the weights remain fixed. Another input produces different states through the same calculation rules.

Token: A small piece of text processed by the model. It may be a word, part of a word, or a few characters.

Vector: An ordered sequence of numbers. The model uses one to represent a token’s current state.

Connecting A and B Reveals the Whole Model

The entrance can turn text into numbers. Where do those numbers go next? I drew the entire route before examining each stop.

This route is a Transformer. Attention and a feed-forward network form a layer, and many layers connect in sequence. Blue A is attention; gold B is the feed-forward network. The first layer feeds the second, the second feeds the third, and so on to the output.

Our task is to continue text, so the backbone is a decoder-only Transformer. The prompt and text already generated follow the same path, providing the basis for predicting the next token. At each position, the model can use that position and the text before it.

The name comes from an earlier translation architecture. In the original paper, an encoder first processes the source language, and a decoder uses the resulting representations to generate a translation. This decoder-only route gives existing text directly to the generation backbone, without a separate text encoder. “Rain outside. I picked up my” becomes the basis for continuing the sentence.

Only three layers are expanded, enough to show the connections. Each layer passes through A, then B. A blue bar is an entire vector. Gold B expands that vector: each dot is one number, and a column of dots together forms a vector. We will follow the circles in the overview to open A and B, keeping their colors and letters consistent.

Transformer: A neural-network architecture organizing attention, feed-forward networks, and other computations into multiple layers.

Decoder-only: A generation backbone that processes a prompt and continues producing text. Attention at each position here reads only that position and earlier ones.

Layer: A group of computations repeated through the model. Every layer in the overview contains blue A and gold B.

Model depth: How many such layers the model contains.

Attention: Blue A in the overview, which retrieves information from earlier text.

Feed-forward network: Gold B in the overview, which further processes the current state.

How Does the End of the Sentence Learn That It Is Raining?

“I picked up my” could lead to many objects. To continue with “umbrella,” the final position needs the earlier rain information. Blue A now has a concrete problem: with so much preceding text, which information should it retrieve this time?

A begins with a comparison. The final position has its own numbers, as does every earlier position. Learned transformations compare these representations and produce relevance scores. The scores become proportions adding up to 100%, determining how to mix information retrieved from each position.

The first round in the diagram assigns 40% to “Rain,” 15% to “picked,” and 45% to the remaining positions. Following the 40% shows what it does: the vector retrieved from “Rain,” after transformation by learned weights, has every component multiplied by 0.40. Other positions receive their own proportions. Adding corresponding components produces a mixed vector.

The blue bar is still a sequence of numbers. The 40% is one number multiplying each component in that sequence. After mixing, the final position has received information from the preceding text and can update its state. These proportions make the process visible; another input or layer produces new proportions.

After the first A, B processes the result. Meanwhile, earlier positions are updating too. At the second layer, both the final position and the preceding text carry updated numbers, so A compares and mixes them again. The second-round proportions in the diagram change accordingly. The network can also perform several such comparisons and mixtures at once, each looking for different relationships. This figure follows one of them.

Attention: A method that compares information at available positions, then retrieves and mixes it according to calculated proportions.

Context: The preceding information available at a position. This generation model can use only the current position and earlier ones.

Attention proportion: How much one position contributes to a mixture. The model calculates it during processing, and it can change with input and layer.

Once the Rain Information Arrives, What Makes It Useful?

A has answered where information comes from. But combining “rain” with “picking something up” still requires processing before it helps predict the next text. I opened gold B in the overview.

Every dot is one number, and a whole column is a vector. Connections carry weights learned during training. An input is multiplied by its connection’s weight; products from several inputs are added, then passed through a transformation to form a new number. Many such connections work together to turn the incoming state into another representation.

The numbers arriving at B already carry context. How B processes the relationship between rain and picking something up depends on its learned weights. Training tests the entire prediction path and adjusts B’s weights along with it, gradually learning transformations that help prediction.

All text positions in the same layer share this B, each sending in its own vector. Different inputs produce different outputs. Here is the earlier distinction again: weights on B’s connections come from training and stay fixed during generation. The 40% and 15% in A are proportions produced by this particular calculation.

Connecting A and B gives a layer its main work: exchange information among positions, then process each position’s state. Another path adds the existing state to the new result, letting later calculations build on information already present. Steps that adjust numerical scale also help keep computation stable. The overview retains A and B as the main route, supported by these additional operations.

Feed-forward network, or FFN: A neural network that transforms input numbers into output numbers using learned weights.

Weight: A learnable number determining how strongly an input affects the next result.

The First Layer Read the Context. Why Does the Second Read It Again?

Following A and B raised another question: if the first layer already retrieved earlier information, why repeat that in the second? Tracing the numbers supplied the answer. The second layer receives different material.

The first layer starts with representations obtained directly from the lookup table. After one round of exchange and processing, the position for “picked” can carry information about “I picked.” When the final position retrieves from it in the second layer, it gets something richer than the initial isolated representation. Its own state has changed too, giving it a different basis for comparison and an opportunity to find further relationships.

With only B afterward, each position could keep processing only the information already retrieved. Keeping A in every layer makes newly processed information available to other positions again. The route becomes an exchange: retrieve, process, retrieve again, process again. The third layer continues from the second, and more layers provide more opportunities to combine information.

The layers look similar, but each has its own weights. What a particular layer learns emerges through training. The design choices directly available are layer count, vector width, B’s size, and the connections.

Now the 500B on the page has a place: it is the total of roughly 500 billion learnable parameters across the model. More layers add parameters, as do wider layers and larger B networks. 500B alone does not specify a unique network. Depth, width, and capacity in each component still need to be allocated in one parameter inventory.

Parameter: A number adjusted during training and retained in the trained model.

State: Numbers produced while processing a particular input. The same model uses the same parameters for different sentences but produces different states.

After All Those Layers, Where Does “Umbrella” Appear?

The final position leaves the last layer still carrying a vector. Another transformation at the output scores candidate tokens in the vocabulary, then converts scores to probabilities. If the model has learned these relationships well, continuations associated with “umbrella” can receive a high probability in this context.

The generation rule selects the next token from those probabilities, appends it, and continues predicting. One word may span several tokens, and a sentence grows piece by piece. Actual systems retain some earlier computation so later steps can reuse it.

The route on the page finally connects end to end. Text becomes numbers through a lookup; A retrieves information; B processes it; many layers follow; the output scores the next piece of text. Looking back from that exit, the 500B parameters now have concrete locations and jobs.

Vocabulary: The set of tokens the model can output.

Output probability: The likelihood the model assigns to each candidate token given the preceding text.

A Public Model Gives the Diagram Concrete Dimensions

The route worked on the diagram. I wanted a public design to compare: how many layers did a real model have, and how were they arranged? I opened Meta’s Muse Glimmer, released in August 2026. Its dense language backbone has 52 layers, with roughly 30B parameters overall, aimed at local deployment. It is smaller than our target, but enough to connect this diagram with a public design.

It also uses different attention ranges: some layers look nearby, while others reach farther back. That variation is now easy to locate. It changes which positions blue A may read. Layer order, reading range, and network width all map to specific places on the drawing.

Dense: Here, every token uses the same complete feed-forward network in each layer.

MoE: Mixture of Experts. It replaces B with several selectable networks, letting a token call only some of them.

Can the Network Grow Without Computing All of It Every Time?

Once the diagram was clear, another cost question emerged. In a dense model, every token passes through the complete B in every layer. Enlarging that network adds parameter capacity and computation. Our 500B engineering baseline follows this dense route, and the machine and training plans continue around it.

But I wanted to examine another choice: could we keep more parameter capacity while calling only part of it for each token? Gold B is the natural place to start. Replace one network with several experts and let a small network choose a few, and we arrive at MoE.

The next article follows that question. We keep the overview, open B, and examine expert selection, how avoided computation reaches the GPUs, and what extra data movement it requires. Reading DeepSeek-V4.1-Flash-1017’s architecture in that same location makes it easier to see what changed and how to compare the two routes.

A closer look: why does B need a transformation in the middle?

Following B’s connections raises one more question. If every operation in the middle is multiplication and addition, multiple layers can collapse into a single calculation of the same kind. That limits what extra depth can express. Adding a nonlinear transformation lets the network represent more complex relationships. Actual feed-forward networks may also include branches and gates. The diagram keeps input, processing, and output visible so the dots and weights are easy to relate.

The Transformer combines attention and feed-forward networks in repeated layers. The original paper presents the basic structure. The modern dense example used here comes from Meta’s public model description. The figures reduce the number of layers and dots; complete models follow their own configurations.