[Training a 500B LLM from Scratch] 2/25: How Many GPUs Do We Need?

The B200 price was on the spreadsheet, but the quantity was still blank. Turning the previous round of GPU shopping into an order required one more answer: how many servers?

I could enter a thousand GPUs, or two thousand. Each change would move the price, while the end date remained unknown. A quantity needed a reason, and that meant first working out how much work the machines had to do.

I divided the sheet into three steps: how much material the 500B model would study, how much computation that learning required, and how machine speed translated into time. Then GPU count could be chosen alongside how long we were willing to wait.

Why Start a 500B Model with 10T Training Tokens?

The model column already said 500B: 500 billion numbers adjusted during training. This round starts with a dense structure, where the main parameters participate in processing the input. Its workload can be added up layer by layer.

Parameter count told me how large the model was. Reading a page and reading a library clearly required different amounts of computation. The material column needed a number too.

A training system splits text into small pieces called tokens. A token can be a character, a word, or only part of a word. Here, 1T tokens means one trillion of these pieces. The more tokens the model reads, the more computation it usually needs.

To find a starting point for that blank cell, I looked at research on model size and training volume. One common starting ratio is about 20 training tokens for each parameter. This ratio comes from earlier research into how to allocate training compute: for a given amount of compute, model size and learning material have a relatively efficient balance.

A scaling-law study from May 2026 reviews this approximate 20-to-1 rule and also shows that directly extending a fixed ratio across scales and training conditions can introduce errors. We will therefore use it as the starting point for the first design, then adjust it with our own data and experiments.

The first objective was to complete a 500B base model, evaluate its quality, and learn what the engineering system could sustain. I started with the 20-to-1 ratio and wrote the calculation beside it:

500B × 20 = 10,000B tokens = 10T tokens

In other words, we plan ten trillion training tokens for 500 billion parameters. The ratio of 20 is an empirical allocation rule. It does not mean each parameter reads its own 20 tokens. The whole model processes the material together.

Once 10T was in the sheet, another choice appeared: would reading more help? Improvements on the target tasks would have to answer that. Extending to 20T or more needs sufficient benefit from the extra computation. Some projects instead aim to train a smaller model on more material to reduce its later serving cost. That produces a different combination of model size and data volume. This round will focus on the 500B model and its first complete pretraining run.

10T settled the workload account, but it did not put material on disk. The data inventory still had to explain how much distinct content there is, which material needs to be repeated, and how often. Reading the same material twice counts twice toward the training volume but may not produce twice the improvement. In its 2026 introduction to Muse Spark, Meta also described using a series of smaller model experiments to establish scaling behavior. Our experiments need to answer the same question: how much improvement does adding this material actually bring?

The sheet now had two inputs: a dense 500B model and 10T cumulative tokens. There was finally a way to calculate the quantity of machines.

How Much Computation Does Reading All This Material Require?

Layer: A unit that takes the previous layer’s result and performs another stage of processing.

Training step: One learning cycle: read a batch of material, calculate predictions and errors, and update the parameters.

To shorten the two large numbers, I used N for the parameter count, 500 billion, and D for cumulative training volume, ten trillion tokens. Next came converting N and D into operations the machines could perform.

I followed one learning cycle. The model first makes a prediction, then traces the error backward to determine how the parameters should change. The forward prediction takes about 2N operations, the backward calculation about 4N, giving 6N altogether. Processing D tokens gives about 6ND of main computation. N already sums the parameters in every layer, so their work is included.

Follow the derivation: why six, and where is every layer counted?

The model has many layers, and every layer does need computation. N already adds up the parameters across all layers of the model. As a simplified example, suppose there are 100 layers with 5 billion parameters each. The total is 100 × 5 billion = 500 billion = 500B. This is an illustration, not an architecture already chosen for this article. Actual layers can also have different parameter counts.

As we will explain below, the main training computation for one token in an ordinary dense model can be approximated as six times each layer's parameter count. Adding layer 1, layer 2, and all the remaining layers gives 6 × layer 1 parameters + 6 × layer 2 parameters + … = 6 × total model parameters = 6N. Every layer's work is therefore included. N already accounts for the number of layers, so multiplying by 100 layers again would count the work twice.

One floating-point addition or multiplication counts as 1 FLOP. Multiplying two numbers and adding the result to a running total counts as 2 FLOPs. A GPU can execute the multiply and add as one fused instruction, but we still count two operations. A single training step contains a large number of these operations.

Start with how the model calculates a result. Multiplying inputs by their weights and adding them together is one of the most common computations in a neural network. Suppose the three inputs are 2, 4, and 6, with weights of 0.2, 0.3, and 0.5. The result is:

2 × 0.2 + 4 × 0.3 + 6 × 0.5 = 0.4 + 1.2 + 3 = 4.6

This takes three multiplications and two additions, or five floating-point operations. Extend the sum to thousands of terms, and the counts of multiplications and additions become almost equal. For large computations, we therefore usually count one multiply and one add per term, or two operations, ignoring the one fewer addition at the start of each sum.

Weight: A type of model parameter that controls how strongly an input affects a result.

Transformer: A common language-model structure in which each layer connects information from the context and then processes that representation. The architecture article will explain the inside of a layer.

Attention: The computation that selects which information in the context is useful for the current content.

Arrange many of these weighted sums together, and we have a matrix multiplication. Much of a dense Transformer's computation happens in these matrices. Adding up the work in the main weight matrices gives about 2N operations per token for the forward pass. Forward means calculating from the input through to the prediction. In practice, the GPU groups many tokens into matrices and processes them together.

Training also needs to determine where the prediction went wrong and how the parameters should change. Tracing backward from the result is called backpropagation. For the main linear computations, it requires two more kinds of matrix multiplication.

One calculates weight gradients: how changing each weight slightly affects the final error. The other calculates input gradients, passing the error signal to earlier layers. The main computation in each is comparable to the forward pass, at about 2N. Together, the backward pass takes about 4N. NVIDIA's guide to linear layers lists these three computations separately.

The main training computation for one token is therefore:

forward 2N + backward 4N ≈ 6N

Processing D tokens in total gives total compute ≈ 6ND. This basic estimate has been used for years. The compute appendix of the 2022 Chinchilla paper presents both a more detailed breakdown and the 6ND approximation. It estimates the dominant matrix computations in a dense Transformer, without requiring every small operation to follow the same ratio exactly.

The figure first adds the parameters across layers to get N, then adds the forward and backward computation per token, and finally multiplies by the total token count D. The 100 layers with 5B parameters each are only an illustration; layers can differ in size. The formula approximates the main matrix operations.

Following the arrows in the figure, I substituted this plan’s numbers. First, one token:

6 × 500,000,000,000 = 3,000,000,000,000 floating-point operations

That is about three trillion operations per token. Multiply by the cumulative ten trillion tokens:

3,000,000,000,000 × 10,000,000,000,000 = 3 × 10²⁵ floating-point operations

This explains 6 × 500,000,000,000 × 10,000,000,000,000. In scientific notation, multiplying 3 × 10¹² by 1 × 10¹³ gives a leading number of 3 and an exponent of 25, or 3 × 10²⁵.

Actual training does more work. Attention, for example, computes relationships among tokens, and this additional computation grows with text length. Repeated computation to save memory is also outside this basic formula. The “effective speed” used below consistently means the main workload counted as 6ND divided by actual training time. Time spent on extra computation, communication, and waiting therefore appears as a lower effective speed. We must not deduct the same overhead again.

The workload was known. The next cell needed a speed. Lenovo's B200 specifications updated in 2026 give a theoretical dense-compute peak of 2.25 quadrillion operations per second, or 2.25 PFLOP/s, using the 16-bit format called BF16.

The specification gives a peak. Training also spends time moving data, exchanging results, and recomputing to save memory. I entered an effective rate of 1 PFLOP/s per GPU, or 1 × 10¹⁵ operations per second, about 44% of peak. This is the rate used for the current schedule. Once measurements from the target 500B model arrive, replacing this cell updates the whole timetable.

I first tried 5,120 GPUs in the quantity cell, or 640 eight-GPU servers. How long would that leave us waiting? Start by adding their effective speeds.

Together, those 640 machines advance the main computation at:

5,120 × 1 × 10¹⁵ = 5.12 × 10¹⁸ operations/second

Dividing the total workload by that speed gives seconds:

(3 × 10²⁵) ÷ (5.12 × 10¹⁸) = 5,859,375 seconds

The number is easier to read as days, with 86,400 seconds in each:

5,859,375 ÷ 86,400 = 67.8168 days

67.8168 days looked like an answer, but the calendar still had no room for saving progress, periodic evaluation, or recovery. I left 10% of the window for these pauses outside training steps, so training advances during 90% of the time:

67.8168 ÷ 0.90 ≈ 75.352 days

The result became about 75.4 days. I marked 76 calendar days for the main run. Procurement, installation, data preparation, preliminary experiments, and post-training would each need their own schedule.

In this time calculation, communication and waiting within training steps are already included in the effective rate of 1 PFLOP/s per GPU. The extra 10% covers pauses outside those steps. If we directly measure how many tokens a full week actually advances, we can use that week's average speed and avoid deducting the same waiting time twice.

BF16: A format that stores a number in 16 bits, reducing storage and computation needs during training.

FLOP: One floating-point operation, used to count computation work.

FLOP/s: Floating-point operations per second, used to measure computation speed.

PFLOP/s: 10¹⁵ floating-point operations per second, or one quadrillion.

Theoretical peak: The highest computation rate in the hardware specification.

Effective speed: The model’s main computation divided by actual training time, giving an average computation rate.

How Does GPU Count Change the Training Schedule?

Was 76 days an acceptable wait? I changed the GPU count in both directions, leaving the model, material, and effective speed per GPU alone:

Each row uses the same calculation: a dense 500B model, 10T tokens, 1 PFLOP/s per GPU, and a 90% training time fraction. The trade behind 5,120 GPUs becomes clearer. It roughly halves the window compared with 2,560 GPUs. Moving to 10,240 would shorten it further, but double the servers and supporting resources again. I kept 640 main servers and a 76-day run as the initial plan, enough to move procurement and facility planning forward while retaining the expansion comparison.

The extra machines would still have to cooperate. More GPUs can mean more communication and waiting, changing the effective speed per card. Measurements at different scales went onto the acceptance list, to check how much time each expansion actually bought.

To see that risk more clearly, I changed the speed cell too. With 5,120 GPUs delivering only 0.6 PFLOP/s each, the run takes 75.352 ÷ 0.6 ≈ 125.6 days. At 1.4, it takes 75.352 ÷ 1.4 ≈ 53.8 days. Increasing the material from 10T to 20T doubles the work to about 150.7 days at the original speed.

Those changes also clarified what to test: the target 500B model, planned text length, and training settings, with checks on memory, scaling, saving, and recovery. Whether the 76-day schedule held would depend on those results.

Cluster: A group of servers working together.

Main training window: The calendar time reserved for this large learning run, including pauses to save, evaluate, and recover. Procurement, installation, and data preparation have separate schedules.

Plan for 5,120 GPUs and About $293 Million in Training Servers

Quantity and time were finally connected. I returned to the purchase sheet from Part One: 5,120 ÷ 8 = 640 eight-GPU B200 training servers.

On October 27, 2026, Exxact's AMD configuration with 1,536GB of system memory had a base price of $443,278. We selected eight ConnectX-7 training NICs at $1,815 each. The configurator returned $457,798 for the complete server. Its eight B200 GPUs and eight NICs are included, so neither is added again later.

640 × $457,798 = $292,990,720

About $293 million appeared in the subtotal. A 76-day run also needed replacements for failed machines, so I added 20 matching spare servers, a 3.125% maintenance allowance, bringing the total to 660:

660 × $457,798 = $302,146,680

The complete server purchase is about $302 million, with 5,120 active GPUs and 160 spare GPUs, or 5,280 in total. A spare replaces a failed training position; it does not contribute to the training rate calculated here.

Now 640 main servers, 20 spares, 500B, 10T, and 76 days sat on the same sheet. If speed or the data plan changed, following the formulas back would update machines, time, and funding together.

But $302 million bought only the servers. Where would they go, how would they connect, and who would keep them running around the clock? A long stretch of the purchase plan was still empty. The next article fills it in.

System memory: Memory used by the server’s CPU. The 1,536GB figure here refers to system memory.

GPU memory: Memory on the GPU itself. It is already included in the eight B200 GPUs in these configurations.