[Training a 200B LLM from Scratch] 4/25: What Does a Large Language Model Look Like Inside?

In the first three articles, we chose GPUs, estimated training time, and worked through the money. Now we can put the machines aside and look at what they will actually train.

Suppose we give a model this sentence: “It was raining outside, so I picked up my…” How does it continue with “umbrella”? This article follows that sentence from input to output. We will start with a dense model, where each layer uses the same complete network to process every text position. The next article turns one part into several selectable “experts,” creating an MoE model.

Text first becomes a sequence of numbers

The computer splits the sentence into tokens, finds an ID for each piece, and uses that ID to look up a row of numbers. Text now has a form that can enter a calculation. Each position has its own row of numbers, which changes as the model processes it.

Follow the same position at the end of the sentence. The diagram uses “my” and shows just three values. The lookup gives [0.2, -0.4, 0.7]. Blue A brings in earlier text, including “Rain outside,” and the state becomes [0.5, 0.1, 0.6]. Gold B processes it again, producing [0.8, -0.2, 0.9]. Those values pass unchanged into the next layer, where calculation continues.

Read the shapes first: each small cell is one number, and the three cells together form one vector. Three illustrative values make the changes visible; a real model usually represents a position with thousands of values. A brings in preceding text; B processes the resulting values. We will open each part later in the article.

What changes here is the state produced while reading this sentence. The weights learned during training stay fixed during this generation. Think of the weights as the learned way of calculating, and the state as the result of this particular calculation. A different sentence gives the same calculation method different inputs and produces different results.

Token: A small piece of text that a model processes. It can be a character, part of a word, or several characters.

Vector: An ordered sequence of numbers. The model uses one to represent a token's current state.

Look at the whole map, then open one layer

We can now give the whole map a name: Transformer. It is a way of organizing a neural network, combining attention and a feed-forward network into layers and connecting those layers. Attention is one part; Transformer names the overall architecture.

This article draws a decoder-only Transformer for continuing text. It uses a decoder, the text-generating part, as its backbone. The prompt and text already generated follow the same path to predict the next token. In the original translation architecture, an encoder first turns the source-language sentence into representations, and a decoder uses those representations to generate the translation. The decoder-only structure here sends the available text directly through the generating backbone, without a separate text encoder. To predict what comes next, it uses the text already available, so each position reads only itself and earlier positions.

The diagram maps that path. After text becomes numbers, it passes through layer 1, layer 2, layer 3, and onward to the exit. The drawing opens only three layers so that the connections remain clear.

Each layer follows a similar order: blue A, then gold B. The output of layer 1 becomes the input to layer 2, which passes its result to layer 3. We have marked A and B in the first layer. The following two diagrams enlarge those exact parts. The colors and letters always refer to the same places. A blue strip represents the vector carried by one text position. Gold B opens a vector: each small circle is one number inside it. B shows only a few components; the real network is much wider.

Transformer: A neural network architecture that organizes attention, feed-forward networks, and related calculations into layers.

Decoder-only: A structure that uses the text-generating backbone to process a prompt and continue it. Here, attention at each position can access that position and earlier ones.

Layer: A group of computations that repeats through the model. Each layer in the overview contains blue A and gold B.

Model depth: The number of these layers in the model.

Attention: Blue A in the overview, which retrieves information from preceding text. The next section enlarges this part.

Feed-forward network: Gold B in the overview, which processes the current state further. The later B enlargement shows its connections.

Zoom into A: how does “raining outside” help?

At the end of the sentence, the model has already seen “It was raining outside, so I picked up my.” To continue, the final position needs clues from the preceding text. Follow attention through three actions: compare, assign proportions, and retrieve information.

First, compare. The final position has a sequence of numbers, and so does every earlier position. Using a method learned during training, the model compares the final position's representation with those at the available positions and calculates a relevance score for each. A different sentence changes the inputs to this comparison and can change the resulting scores.

Next, assign proportions. The scores become proportions that add up to 100%. These determine how much information to take from each position in this particular combination. In the diagram's English example, the first round assigns 40% to “Rain,” 15% to “picked,” and a total of 45% to the other positions. These proportions are teaching examples that make the process visible.

Finally, retrieve the information. A 40% share for “Rain” means multiplying the sequence retrieved from that position by 0.40. This retrieved representation is a vector transformed using learned weights. The other positions contribute their sequences multiplied by their respective shares. Adding these results produces a mixture of information from the preceding text, which updates the current state. A larger share gives that position more influence in this particular mixture. The diagram separates the two: a blue segmented strip is a complete vector, while the 40% beside it is one number that multiplies every component in that strip. The mixed result is still a vector.

The diagram follows the final position, “my,” throughout. After the first A, B processes the result. The other positions update their own states too. Layer 2 therefore receives a new set of numbers. It compares them again and calculates another set of proportions. A position can retrieve weather clues in one round, then use information already combined in the previous layer to examine further relationships. Training determines the actual division of work.

A layer's attention can also run several groups of comparison and mixing in parallel. The diagram opens one group so that we can see how it brings preceding text into the current position.

Attention: A method that compares information across positions, then retrieves and combines it in calculated proportions.

Context: The preceding text available to the current position. In the generation model discussed here, a position can use information from itself and earlier positions.

Attention proportion: How much a position contributes to a particular mixture. The model calculates it as it runs, and it can vary with the input and the layer.

Zoom into B: how is that information processed?

Now enlarge gold B in the overview. We have opened one vector: each circle represents one number, and a whole column of circles represents a vector. Only a few components are drawn to keep the connections clear. Each connection has a weight learned during training. When a circle receives several inputs, it multiplies each input by its weight, adds the results, and applies a transformation to produce a new value. Many connected calculations can turn the current state into another representation.

B receives a state that already carries context. How it responds to the connection between rain and picking something up depends on its learned weights. Training repeatedly adjusts those weights so that the whole model becomes better at predicting the text that follows.

B processes each text position separately, using the same weights within that layer. Different positions provide different input values, so they can produce different results. The weights on these connections come from training and stay fixed during generation. The percentages in A are mixing proportions calculated while processing this particular sentence. A lets positions exchange information. B then processes the state at each position. Together, they perform the main work of one layer.

Connections inside the layer also carry the existing state forward and add it to new results. The next step therefore receives both the new processing result and information already developed earlier. Real layers also adjust the scale of the numbers to keep calculation stable. The overview focuses on A and B to keep the main path clear.

Feed-forward network (FFN): A neural network that transforms input values into output values using learned weights.

Weight: A value that controls how much an input affects the next result. Weights are one kind of learnable parameter.

What do the second and third layers do?

The first layer receives the initial representations looked up for the text. The second receives results that the first has already enriched with context and processed. The third continues from the second layer's results. Each added layer provides another step for combining and processing information.

Why does layer 2 need A again? It now has combinations of information that layer 1 did not have. For example, an earlier layer can give the “picked” position information about “I picked.” When the final position retrieves information from it in a later layer, it receives something richer than the original isolated text representation. The earlier positions have changed, and the final position's own state has changed too. Comparing these new representations can reveal further connections.

A exchanges information between positions; B processes each position separately. With only one A at the beginning followed by B alone, later calculations could only process the information already retrieved at each position. The Transformer structure chosen for this article puts A and B in every layer. They alternate: exchange, process, and exchange again using the processed results. Layer count determines how many rounds of this interaction can take place.

A and B look similar across layers, but each layer stores its own weights. The same connection pattern can perform different calculations in different layers. Their division of work develops during training. When designing the network, we choose the number of layers, the width of the number sequence each handles, and the connections between them.

This also gives 200B a concrete meaning: roughly 200 billion learnable parameters distributed through the model. Layer count is one design choice. Increasing a layer's width or enlarging the network inside B also adds parameters. The 200B figure alone therefore leaves several choices to make before we have a complete architecture.

Parameter: A value adjusted during training and stored in the model afterward.

State: Values produced while processing a particular input. The same model uses the same parameters for different sentences, while producing different states.

At the exit, how does the model generate “umbrella”?

After the final layer, the state at the end of the sentence is still a row of numbers. The output calculation turns it into a score for every candidate token in the vocabulary, then converts those scores into probabilities. If the model has learned these relationships well, a continuation associated with “umbrella” may receive a high probability in this context.

The model selects the next token according to its generation settings, appends it to the text, and continues predicting. A word may require several tokens, and a complete response repeats the process many times. In practice, the system saves some results from earlier text for later steps to reuse.

We can now follow the full path: text becomes numbers, A retrieves context, B processes the state, layers pass results onward, and the exit scores candidates and selects the next small piece of text.

Vocabulary: All tokens the model can output.

Output probability: The likelihood the model assigns to a candidate token given the preceding text.

Meta still uses this dense structure

There is a current example. Meta's Muse Glimmer, released in August 2026, uses a dense language backbone with 52 layers. At roughly 30B parameters overall, it is designed for local deployment. It gives us a publicly documented design smaller than our 200B target.

It also varies the reach of attention. Some layers read nearby text, while others can read farther back. On our map, this changes which positions blue A can access. Once A and B are clear, a design choice like this has a visible place in the network.

Dense: Here, each token uses the same complete feed-forward network in every layer.

MoE: Mixture of experts. It replaces B with several selectable networks, so that a token calls only a subset; the next article explains how.

Which part will we change next?

This article follows the complete path through a dense model. For our 200B project, it provides a clear structural baseline: the same feed-forward network processes each token, tying parameter scale closely to computation per step. Layer count, width, and the capacity allocated to A and B all belong in one parameter budget.

The next article keeps this map and changes gold B. We will turn one feed-forward network into several experts, explain who selects them, why that can reduce computation per call, and what extra data must travel between GPUs. We will also examine DeepSeek-V4.1-Flash-1017's specific choices at that point.

The two articles answer connected questions. This one explains how a complete model works. The next explains where MoE changes it and why we might choose that approach. Once the two routes are clear, we can select a specific configuration for our own 200B model.

A closer look: why does B need a transformation in the middle?

If every step only multiplies and adds, several such layers can still be combined into one calculation of the same kind. A nonlinear transformation lets the network represent more complex relationships. Real feed-forward networks can also include branches and gates. The drawing preserves the main input, processing, and output path to make the relationship between values and weights clear.

The Transformer combines attention and feed-forward networks into repeating layers. The original paper describes that foundation. The modern dense example here comes from Meta's public model card. These diagrams are teaching illustrations with simplified layer and circle counts.