[Training a 200B LLM from Scratch] 5/25: How Does MoE Use Only Some of Its Parameters for Each Calculation?

In the previous article, we followed “Rain outside. I picked up my…” through a model. In each layer, blue A retrieved information from the preceding text, and gold B processed the current state. With many layers connected, the model could score the text that might come next.

Now we want the model to hold more of what it has learned. One way is to add parameters. But in the dense model from the previous article, every token passes through the complete gold B in each layer. Making that network larger also adds computation every time an input passes through it. The model gains capacity, and the compute needed to train it and use it to answer questions grows too.

MoE addresses this problem: how can a model hold more parameters while controlling the work required for each calculation? It turns gold B into several expert networks and selects only a few for each token. Blue A still retrieves information from the preceding text. The selected experts process it, and their result moves to the next layer. We can keep adding experts to expand capacity while each token passes through only a small part of the model. This is the benefit of MoE: we can design “how large the whole model is” and “how much it computes each time” separately. A similar compute budget per token can support more parameters. Training and evaluation show how much that extra capacity improves the model. GPUs still need to store those parameters. Whether less computation takes less time also depends on how the work is assigned to the machines.

This article follows the same sentence. We will first watch one token select its experts, then place that path on GPUs, and finally return to our 200B project to calculate what changes.

Keep the Overview, Open Gold B

In the dense model from Part Four, each layer has one B. Whether the input is “rain” or “picked up,” it goes through that network. Different inputs produce different outputs; the calculation uses the same set of weights.

MoE expands this block into several networks. Each has its own weights, receives a vector, and produces a vector. We call each such network an expert. Here, an expert is the feed-forward network we enlarged in the previous article. Inside it, numbers still travel through weighted connections to produce new numbers.

The diagrams keep blue A and gold B. The first diagram marks B, and the next opens that exact location. A segmented strip represents a vector, with one number in each cell. Boxes marked E1 and E2 represent whole expert networks, each of which can contain many parameters.

The word “expert” can make us picture a language teacher, a mathematics teacher, and a programming teacher. Training usually starts with similar structures and different initial weights, then lets the data gradually produce a division of work. An expert may respond to certain text features, or perform computations that are hard to describe with the name of a subject. We will use labels such as E1 and E2 and follow the numbers to see how they work.

MoE (Mixture of Experts): a structure that places several expert networks in part of a model and selects only some of them for each token.

Expert: here, a feed-forward network with its own weights that receives and produces vectors.

Dense model: in this comparison, every token uses the same complete feed-forward network in each layer.

Who Chooses the Experts?

Keep watching the end of the sentence. In the English diagrams, this position is “my.” After blue A, its vector already carries information from the preceding text. At the entrance to B, a small network reads this vector and scores the experts. This small network is the router.

The diagram starts with four experts and selects two each time. For this input, the router selects E1 and E3, assigning their results mixing weights of 75% and 25%. The weights of E2 and E4 still exist; this token does not use them on this pass. The selection can change for another token or another layer.

Both selected experts receive the same complete input vector and calculate with their own weights. Suppose E1 produces [0.8, 0.2, -0.4] and E3 produces [0.4, 0.6, 0.8]. We combine the two results in the assigned proportions:

0.75 × [0.8, 0.2, -0.4] + 0.25 × [0.4, 0.6, 0.8] = [0.7, 0.3, -0.1]

The first cell is 0.75 × 0.8 + 0.25 × 0.4 = 0.7. We calculate each remaining cell in the same way. The result is still a vector, with the same width as the expert outputs. This is the result B produces on this pass. It then follows the layer's connections onward.

Keep three things separate when reading this diagram: the segmented strip is a sequence of numbers, the expert box is a network, and 75% is one number. That number multiplies the entire vector, which means every component in it. The arrows show where data goes. The learned connections inside each expert are not expanded here.

These two proportions are an example chosen to make the arithmetic easy to see. Each model defines its scoring, selection, and normalization rules; some also apply a scaling factor. The central process remains easy to follow: read the current vector, select a few experts, calculate separately, and combine their results.

This also shows how MoE and attention divide the work. The proportions in A determine which text positions supply information. The routing in B determines which expert networks process the current position. Both may use weighted sums, but they select different things.

Router: a small network that reads the current state, scores experts, and decides which ones to use.

Top-k: selecting the k highest-scoring candidates from a group. In the diagram, k is 2.

Routing weight: the proportion used to mix an expert's output, determined by the current input and routing method.

Shared Experts: A Path Every Token Takes

We can add another arrangement: keep one expert that every token passes through to handle commonly needed work, alongside a set of experts selected as needed. The first is a shared expert. The others are routed experts.

A token entering B then passes through the shared expert and the selected routed experts, and their results are combined. Both kinds receive and produce vectors. The difference is which inputs call them and how their results are combined.

This gives training another way to divide the work. Common processing can remain on the shared path, while the routed experts learn different processing methods. Their behavior after training shows what they actually learned. Their names describe how they are connected and called.

All text positions in a layer use that layer's expert pool. The next layer has its own experts and router. The selection depends on the token's current state and the layer it has reached.

The training objective is still to predict the following text better. The prediction error follows the paths involved in the calculation and is used to adjust the learnable weights of the experts and router. As many different inputs repeatedly pass through, the experts' processing and the router's choices learn together.

Shared expert: in MoE designs that include one, an expert network called for every token in the layer.

Routed expert: an expert network selected by the router for the current token.

The Model Is Still 200B. Why Can Each Pass Compute Less?

We can now separate two numbers. Total parameters count all the learnable numbers the model stores. Active parameters count how many a token calls along its current path. MoE separates these numbers, allowing the model to store more weights while controlling the network size each token passes through.

Let us calculate with a rounded 200B parameter ledger. We assign a rounded 40B to the major matrix computations on common paths, including attention projections and shared experts. The remaining 160B is distributed among the routed experts across all layers. Each layer has 16 equally sized routed experts and selects 2 each time.

This rounded ledger counts those major matrices first. Details such as input lookups will be checked when the complete configuration is defined. With the same selection ratio in every layer, the routed parameters called per token are:

160B × 2 ÷ 16 = 20B

Adding the common part gives:

40B + 20B = 60B

The 160B is the total across all layers' routed experts. The sixteen cells in the diagram represent their common selection ratio. Each layer in the actual model has its own expert pool. Block areas show parameter counts, not vectors or numbers of GPUs.

The model still stores 200B parameters, while a token mainly passes through about 60B. The major multiply-add work on these weights falls with the amount called. Compared with using all 200B for every token, 60B represents roughly 30% of this part of the computation.

The saved computation creates room for more model capacity. The machines also perform attention, select experts, move data, and, during training, save and update parameters. Longer contexts also change the cost of attention itself. Adding these tasks gives the actual time required for training.

Memory needs a separate ledger. Experts unselected by the current token still need to be stored because another token may use them next. A training batch contains many tokens. Together they may call many or even all experts, with training signals updating weights along their respective paths. Training also stores state for these parameters. MoE therefore first changes how much computation each token calls. It does not directly shrink a 200B model's storage requirement to that of a 60B model.

Total parameters: the number of all learnable parameters stored by the model.

Active parameters: the parameters a token calls along its current computation path. When comparing models, check whether the count includes shared parts and which execution path it describes.

GPU memory: memory on the GPU that holds the model, current data, and intermediate results.

If Four Experts Compute at Once, Is Selecting Two Faster?

Start with a specific example: one token, with four experts each placed on a separate GPU. All four GPUs have enough computing capacity, and each expert needs 1 millisecond to process this token. The input has already arrived. For now, we count only the expert computation step.

Selecting two of four is not faster in this example. The four experts could already finish at the same time. Leaving two experts unselected leaves the other two calculating for 1 millisecond each. The total amount of work falls, but the token's wait for this step stays the same. The other two GPUs really are idle, while still storing their experts' weights.

When can time fall? When the same hardware originally needs several rounds to finish the work, less work can reduce the number of rounds. A GPU has a finite number of compute units, and large matrices are usually split into smaller tiles processed over multiple rounds. NVIDIA's matrix multiplication guide describes this arrangement. Parameters are stored in memory, and compute units repeatedly read and process them. Each parameter does not have a dedicated compute unit.

We therefore need to ask two questions separately: how long one token waits for its result, and how long a whole batch takes to finish. MoE sends each token through only some experts, first reducing the amount of computation. Whether this makes the current token faster or finishes the batch sooner depends on whether the original work could already run at once, and how much time goes to transfers and waiting. Widening the view to a batch of inputs shows another situation.

Compute unit: a part inside a GPU that performs calculations, repeatedly processing different data and parameters.

Latency: the time from starting a request to receiving its result. The table counts only the expert computation step.

Throughput: the amount of work completed per unit of time, such as tokens processed per second.

Putting Experts on Different GPUs Introduces New Waiting

E1 and E3 sit close together in the earlier diagram. In an actual model, the experts may be on different GPUs. After the router selects them, the system sends the input vector to their GPUs, runs the experts, and returns their outputs for combination.

This is still the same B. E1 is on GPU 1 and E3 is on GPU 2. The token's input travels to both places. Both outputs must return before the weighted combination can finish. Other text positions in the model are doing similar work.

Now widen the view from one token to a batch of data. The current token selects E1 and E3, while other tokens in the batch may select E2 and E4. An expert that did not receive this token may be processing other tokens. One GPU can also store several experts, with the same compute units taking turns on their work.

Consider a simple assignment example: four GPUs each hold one expert, and a batch contains 100 tokens. If every token passes through all four experts, there are 400 expert calls, with 100 handled by each GPU. Selecting two experts per token gives 200 calls. With an even distribution, each GPU handles 50. Its workload falls from 100 calls to 50. If the original work needed multiple rounds, reducing it can shorten the batch's completion time. If all 100 calls could already finish at once, 50 may be no faster. Actual performance also depends on how fully each GPU is used and how long data distribution takes. This compares two calling patterns for the same set of experts; model quality needs a separate evaluation.

If most tokens select E1, however, GPU 1 develops a queue while other GPUs may have spare capacity. With a small batch in particular, only a few GPUs may be busy while the others hold their weights and wait for another round. Here, the saved work may mostly become idle time, and the batch may not finish sooner. Training needs to assign suitable amounts of work to experts while preserving the router's ability to select experts for each input.

This is why MoE papers spend so much space on load balancing and communication. The expert networks save computation, while task assignment, cross-GPU transfers, and waiting add work. A useful MoE system needs to handle both sides well.

NVIDIA's Megatron Bridge performance guide discusses expert placement, grouped computation, and communication together. Engineers pack vectors bound for the same GPU into batches, try to overlap transfers with computation, and place experts according to device connections. We will examine how experts fit into our B200 servers and which data crosses the switches in the articles on parallel training and networking. For now, keep one direct way to judge the system: alongside tokens per second, look at how much work each expert receives and how long GPUs wait for data.

Batch: a group of training data processed together, usually containing many tokens.

Expert parallelism: placing different experts on different devices and sending inputs to them according to routing decisions.

Communication: here, transferring inputs, outputs, and other data between GPUs.

Load balancing: distributing work appropriately among experts and devices to reduce situations where some have long queues while others sit idle.

Put DeepSeek's Actual Design Back into This Diagram

Once this path is clear, the numbers in recent models have a place to go. DeepSeek's V4.1-Flash-1017, released on October 17, 2026, provides a concrete example. Its public configuration sets 384 routed experts in each MoE layer, selects 6 per token, and retains 1 shared expert.

Mapped to B above, the four example experts become 384, “select two” becomes “select six,” and an always-used shared path is added. Each expert remains a feed-forward network. There are more experts, but we can still follow the basic data flow through the same diagram.

Read the shared and routed parts separately. Selecting six describes the choice among the 384 routed experts. Attention, the shared expert, and other components still work along their respective paths. This is why the 200B example first separated out 40B of common parameters: those tasks need to be counted too.

Meta also retained both approaches in 2026. Muse Glimmer 30B, introduced in Part Four, uses a dense language backbone for local operation. Muse Spark Open, released on October 15, uses MoE, with an official count of 816B total and 42B active parameters. A team chooses different structures according to capacity, speed, and deployment environment. These are real designs we can study. Choosing one for this new company requires comparing them against our own task.

A closer look at DeepSeek's 552B, 196B, 8B, and 16B figures

The technical report lists a 552B language backbone plus a 196B Engram memory module. Engram stores and retrieves information in another way. We will return to it when discussing memory.

The report's 8B and 16B figures describe the inference paths for reading input and generating step by step, respectively. They show how this architecture controls inference computation. Training from scratch requires counting the actual training path. To calculate the MoE selection ratio in this article, identify the routed expert portion first, then add the common paths. This avoids multiplying the entire backbone by 6/384.

What Should Our 200B Project Choose Next?

The 2,048 B200 GPUs, roughly 31 days of main training, and the budget in Part Three were based on a dense 200B model processing 4T tokens in total. They remain a complete baseline for comparison. Adding an MoE candidate gives us another way to train a model on this equipment.

I would first specify three things for the candidate: how many parameters it stores, how many each token calls, and how each layer's experts are placed on devices. The 200B ledger with roughly 60B active parameters provides a starting point for the compute scale. For an actual implementation, the layer count, vector width, expert sizes, and shared parts all need to be calculated into the parameter inventory.

Then run comparisons using the same data and evaluation methods. We want to see whether MoE learns more with the same training compute investment, how much saved computation becomes faster training across multiple GPUs, and whether every expert receives enough training. Measured throughput at the target scale then feeds back into the training schedule, GPU count, and cost.

This article supports one initial decision: put MoE in the 200B project's formal candidate set and test it alongside the dense baseline. It gives us a chance to design “how much the model holds” and “how much each token computes” separately. Whether it is worth adopting depends on the capability gained from that extra capacity and how quickly our machines complete the entire path.