[Training a 500B LLM from Scratch] 5/25: How Does MoE Compute with Only Some of Its Parameters?

The previous diagram was still on the table: blue A retrieved context, gold B processed it, and layer after layer led to the next piece of text. I looked again at B. In the dense 500B model, every token makes that whole network work in every layer. More capacity brings more computation each time a token passes through.

Could the model hold more of what it had learned while using only the part needed this time? Different text brings different inputs. Having all of B participate every time seemed worth reconsidering.

I drew several networks beside B and added a choice at the entrance: send the current token to a few of them, leaving the others for different inputs. That change produces MoE. Total parameters determine how much capacity the model stores; the number of experts selected determines how much computation this token calls. The two numbers can now be planned separately. How much capability the extra capacity adds remains a training and evaluation question. More immediately: would doing less work actually be faster on a GPU?

First, follow the same token into B and see how its route is chosen. Then move the experts onto different GPUs and trace where time can be saved.

Can B Do Something Other Than Compute Everything Each Time?

The original B has one set of weights. “Rain” arrives, and B processes it. “Picked” arrives, and the same B processes that. Different text gives different input vectors, but uses the same network.

The new diagram expands B into several networks. Each has its own weights, takes a vector, and returns a vector. That is an expert. Inside each box are the weighted connections from the previous article: input numbers are computed into new numbers.

A and B remain in their original positions. The circle around B identifies what the next figure opens. A segmented bar is an entire vector; a cell is one number. Each E1 or E2 box is a complete expert network that may contain many parameters.

It would be easy to label them “math,” “coding,” and “language.” I erased those names and used E1 and E2. Training commonly starts with similar structures and different initial weights, and specialization develops through the data. Experts may favor certain features or learn behavior that does not fit neat subject labels. Numbering them makes the actual computation easier to see.

MoE, or Mixture of Experts: A structure with multiple expert networks in part of the model, where each token selects only some of them for computation.

Expert: Here, a feed-forward network with its own weights that receives and returns vectors.

Dense model: In this comparison, each token uses the same complete feed-forward network in every layer.

Who Decides Which Experts to Call?

The final “my” has passed through A and now carries context. Four experts are waiting, but which should receive it? A small network at B’s entrance reads the vector and scores the experts. This is the router.

I first drew a complete route with four experts, selecting two per token. This time it chooses E1 and E3, mixing their results at 75% and 25%. E2 and E4 remain stored in the model but are not called for this token on this pass. Another input or layer may choose differently.

Routing does not split the input in half. E1 and E3 each receive the same complete vector and process it independently. On the page, E1 returns [0.8, 0.2, -0.4] and E3 returns [0.4, 0.6, 0.8]. Their outputs combine component by component:

0.75 × [0.8, 0.2, -0.4] + 0.25 × [0.4, 0.6, 0.8] = [0.7, 0.3, -0.1]

The first component is 0.75 × 0.8 + 0.25 × 0.4 = 0.7. The other two follow the same calculation, leaving a vector of the same width. That becomes B’s output and continues along the layer’s connections.

This is also where the visual distinction matters most. The bar is a sequence of numbers; an expert box is a network; 75% is one number multiplying every component of an output vector. Arrows show where data travels. Learnable connections inside an expert remain inside its box.

Using 75% and 25% makes the result easy to check component by component. A complete model configuration must also specify scoring, Top-k, normalization, and sometimes scaling. The route itself is now clear: read the state, select experts, compute independently, and combine.

A and B both calculate proportions, but they select different things. A chooses text positions from which to retrieve information. B chooses networks to process the current position. Context arrives first, then enters the experts. The model’s overall route remains connected.

Router: A small network that reads the current state, scores experts, and selects which to use.

Top-k: Selecting the k highest-scoring candidates. Here, k equals two.

Routing weight: A proportion used to combine expert outputs, determined by the current input and routing method.

Where Does Work Needed Every Time Belong?

Once the expert pool was drawn, another issue appeared. Many inputs may repeatedly need some common processing, and routing it afresh each time may not be the best arrangement. B can retain a path that every token takes, called a shared expert. The selectively called networks beside it are routed experts.

An input entering B goes to the shared expert along one route and the selected routed experts along another. Each produces a vector, and the model combines them according to its design. Their calling patterns differ; receiving and processing vectors is the same kind of work.

Common processing now has a regular place, while routed experts can develop their own behavior. Their actual specialization must be observed after training. The shared and routed names first tell us which paths each input takes.

Copying the diagram into the next layer also means using that layer’s own experts and router. Positions within one layer share its expert pool; the next layer has different weights. As a token moves through the network, both its state and the pool it encounters change, so its route can change too.

All these choices ultimately face the same test: how well do they predict subsequent text? Error travels back through the paths that participated, adjusting learnable weights in experts and routers. Batch by batch, processing roles and selection behavior develop through training.

Shared expert: An expert called for every token in an MoE layer that uses this design.

Routed expert: An expert selected by the router for the current token.

Why Store 500B but Use Only 150B This Time?

With the route drawn, the parameter account had two numbers. I labeled the columns “total” and “active”: how many learnable numbers the model stores, and how many a token calls on this path.

Start with a round-number 500B architecture account. Assign 100B to the main matrix computations on common paths, including attention projections and shared experts. Put another 400B in routed experts across the layers. Each layer has sixteen equally sized routed experts and selects two.

The routed portion now has a straightforward calculation. With the same selection ratio in every layer, the main matrix parameters called from that 400B are:

400B × 2 ÷ 16 = 50B

The common paths still run every time, so add their 100B:

100B + 50B = 150B

The 400B is the total across routed experts in all layers. Sixteen blocks represent their shared selection ratio; each actual layer still has its own expert pool. Block area means parameter count, not a vector or a number of GPUs. Smaller items such as input lookups remain for the complete configuration inventory.

The model still stores 500B, while a token mainly calls about 150B. Looking only at the main multiply-add work on those weights, 150B is about 30% of 500B. The saved work has a clear source: unselected experts do not compute for this token on this pass.

The training-time column cannot simply become 30%, however. Attention, routing, data movement, and parameter updates continue. Longer context adds attention work too. Time depends on adding these operations back along the whole route.

Memory does not shrink to 150B either. Experts not selected for this token remain stored, ready for later inputs. Many tokens in one batch may together reach many or even all experts. Training also retains the corresponding parameter states. There are now two accounts: how much to store, and how much to compute per token.

Total parameters: All learnable parameters stored by the model.

Active parameters: Parameters called along a token’s current computation path. Comparisons must check whether shared components are included and which execution path is counted.

GPU memory: Memory storing the model, current data, and intermediate computation results.

If Four Experts Can Compute at Once, Why Would Selecting Two Be Faster?

After writing “less computation,” I still had a question. GPUs already run work in parallel. If selecting two experts leaves two other cards idle, where is the time saving? Start with the simplest layout: one expert on each of four GPUs, one token, one millisecond per expert, with the input already delivered.

Both rows take one millisecond. The four experts could already finish together. Selecting two leaves those two taking the same millisecond. Total work falls, but this token’s wait does not. Two cards really are idle, with weights still occupying memory.

That table sent me back to how a GPU computes. It has a finite number of computing units. A large matrix commonly needs tiles processed in multiple rounds. Parameters live in memory, and computing units repeatedly load and process them. Each parameter does not have a dedicated computing unit. NVIDIA’s matrix-multiplication guide explains this tiling. Reducing work that originally required several rounds creates an opportunity to finish in fewer rounds.

Time therefore needs two questions: how soon does one token get its result, and how soon does a whole batch finish? The table answers the first. Training handles many tokens at once. We need to place the whole batch on those four cards and see whether they would still sit idle.

Computing unit: A component inside a GPU that performs operations, repeatedly processing different data and parameters.

Latency: Time from the start of a request to its result. The table counts only expert computation.

Throughput: Work completed per unit of time, such as tokens processed per second.

Put Experts on Different GPUs, and Another Wait Appears

Physical devices add another journey. If experts live on different GPUs, inputs must travel to them after routing, and outputs must return. The small gap between E1 and E3 on paper may require a transfer between cards.

The figure places E1 on GPU 1 and E3 on GPU 2. One input vector travels to both. Their outputs must both return before being combined. Other text positions travel among the devices according to their own routes.

Zoom out. This token goes to E1 and E3; its neighbor may go to E2 and E4. Those latter experts did not receive the first input, but can process the second. One GPU can also store several experts, with its computing units handling their work in turn.

Place a batch of 100 tokens on four cards, one expert per card. Calling all four experts for each token gives 400 calls, or 100 per card. Selecting two gives 200 calls, or 50 per card when balanced. Work that required multiple rounds can now finish sooner. If all 100 calls could already finish at once, reducing them to 50 might still save no time; utilization and transfer waits matter. This calculation compares calls to the same experts. Equivalent capability requires a separate quality evaluation.

Shift more routes toward E1, and another problem appears immediately: GPU 1 develops a queue while others may have nothing to do. Small batches can create similar idleness. Some of the reduced work becomes unused capacity. Turning saved computation into training speed requires a sensible distribution while preserving input-dependent expert choice.

The “load balancing” and “communication” repeated in MoE papers now have concrete locations. Computation is saved inside experts; dispatch, transfer, and waiting are added between them. Together, they determine when the batch finishes.

I continued into NVIDIA’s Megatron Bridge performance guide. It treats expert placement, grouped computation, and communication together: batch vectors heading to the same card, overlap transfers with computation where possible, and arrange experts around device connections. These methods will reach actual B200 machines in the later parallelism and networking work. For now, two items go on the checklist: how much work each expert receives, and how long GPUs wait for data.

Batch: A group of training data processed together, usually containing many tokens.

Expert parallelism: Placing different experts on different devices and routing inputs to them.

Communication: Here, transfers of inputs, outputs, and other data between GPUs.

Load balancing: Distributing work across experts and devices to reduce long queues on one side and idleness on another.

Where Do 384 Experts Fit in the Diagram?

With the diagram in hand, the long configuration for DeepSeek V4.1-Flash-1017, released on October 17, 2026, has a place to land. Its public configuration lists 384 routed experts per MoE layer, six selected per token, and one shared expert.

Expand the four experts in our drawing to 384, change two selections to six, and add the shared path used every time. That gives the broad connections inside this B. Experts remain feed-forward networks, and data still follows selection, dispatch, computation, and combination.

The 6/384 ratio applies only to routed experts. Attention and shared experts still compute, so it cannot multiply the entire model. Separating 100B of common work from our 500B account made room for exactly these always-used paths.

Another public design makes a different trade. Meta Muse Glimmer 30B, used in Part Four, retains a dense language backbone for local deployment. Muse Spark Open, released on October 15, 2026, uses MoE, with 816B total and 42B active parameters listed officially. Even one team chooses different routes for capacity, speed, and deployment conditions. We can learn from those tradeoffs, then put our own workload back into the comparison.

A closer look at DeepSeek’s 552B, 196B, 8B, and 16B

The technical report lists a 552B language backbone and a separate 196B Engram memory module. Engram stores and retrieves information differently; we will return to it when discussing memory.

The report’s 8B and 16B figures correspond to inference paths for reading input and generating incrementally. They describe how that architecture controls inference computation. Pretraining needs a count along the actual training path. For this article’s MoE ratio, locate the routed-expert portion first, then add common paths, avoiding multiplication of the entire backbone by 6/384.

What Do We Choose for Our 500B Project?

The project sheet still says a dense 500B model, 10T cumulative training tokens, 5,120 B200s, and a 76-day main run. That engineering baseline continues. Beside it, the new 500B account with roughly 150B active parameters provides a clear starting point for an MoE comparison.

Before treating MoE as a replacement, I would complete three columns: total parameters stored, parameters called per token, and how each layer’s experts occupy devices. The round-number account then expands into a full inventory of depth, vector width, expert size, and shared components.

The two routes would use the same data and evaluation methods. At equal computation, how much learning does the extra capacity buy? Across multiple cards, how much waiting time is saved? Does every expert receive enough training? Once target-scale throughput is known, duration, GPU count, and cost can be recalculated.

After this circuit, the original question is sharper. MoE offers more stored parameters with less computation per token. Turning that opportunity into time savings requires the whole batch and the hardware to cooperate. One part of the route remains unopened: how do routers, experts, and the earlier Attention learn these jobs from text? Next, we return to “Rain outside. I picked up my umbrella” and follow prediction errors backward to see how they change the numbers in the network.