[Training a 200B LLM from Scratch] 2/25: How Many GPUs Do We Need?

Once we have chosen a GPU, the supplier will ask a very practical question: how many servers do you want?

One thousand GPUs, two thousand, or four thousand? All three sound like the scale of a large model project. All three can quickly turn into a very expensive quotation. We need to trace the question back one step: how much work must the machines do, and how soon do we want the result?

Part One used the B200 as a reference for planning. This article continues that calculation, chooses a configuration for this hypothetical company's first training run, and works out the server cost.

Why Start a 200B Model with 4T Training Tokens?

200B means 200 billion parameters. Think of parameters as numbers the model keeps adjusting while it learns. This article uses a dense model, where the main parameters participate in processing the input. We will fix this architecture first and work out the initial training plan.

With the model size established, we still need to decide how much material to give it.

A training system splits text into small pieces called tokens. A token can be a character, a word, or only part of a word. Here, 1T tokens means one trillion of these pieces. The more tokens the model reads, the more computation it usually needs.

So how much should a 200B model read? A common starting ratio in practice is about 20 training tokens for each parameter. This ratio comes from earlier research into how to allocate training compute: for a given amount of compute, model size and learning material have a relatively efficient balance.

A scaling-law study from May 2026 reviews this approximate 20-to-1 rule and also shows that directly extending a fixed ratio across scales and training conditions can introduce errors. We will therefore use it as the starting point for the first design, then adjust it with our own data and experiments.

This hypothetical company's first objective is to train a 200B base model that can be fully evaluated, establishing a quality and engineering baseline. For that objective, adopting the starting ratio of 20 gives the data volume a clear basis:

200B × 20 = 4,000B tokens = 4T tokens

In other words, we plan four trillion training tokens for 200 billion parameters. The ratio of 20 is an empirical allocation rule. It does not mean each parameter reads its own 20 tokens. The whole model processes the material together.

After 4T, we can continue training. If performance on the target tasks is still improving steadily, we can assess whether extending the run to 8T or more is worthwhile. Another project objective is to train a smaller model on more material to reduce its later serving cost. That produces a different combination of model size and data volume. This round will focus on the 200B model and its first complete pretraining run.

The data team also needs to break the 4T total into specific sources: how much distinct content there is, which material needs to be repeated, and how often. Reading the same material twice counts twice toward the training volume but may not produce twice the improvement. In its 2026 introduction to Muse Spark, Meta also described using a series of smaller model experiments to establish scaling behavior. Our experiments need to answer the same question: how much improvement does adding this material actually bring?

We now have both inputs for the first workload: a dense 200B model and 4T tokens. It is time to bring in the machines.

Step by Step: How Many Days Do 2,048 GPUs Need?

Layer: A unit that takes the previous layer’s result and performs another stage of processing.

Training step: One learning cycle: read a batch of material, calculate predictions and errors, and update the parameters.

First, identify the two quantities in the formula. N is the model's parameter count, which is 200 billion here. D is the total number of tokens processed during training, which is four trillion. We want to estimate how many floating-point operations the model needs to process all this material.

The model has many layers, and every layer does need computation. N already adds up the parameters across all layers of the model. As a simplified example, suppose there are 100 layers with 2 billion parameters each. The total is 100 × 2 billion = 200 billion = 200B. This is an illustration, not an architecture already chosen for this article. Actual layers can also have different parameter counts.

As we will explain below, the main training computation for one token in an ordinary dense model can be approximated as six times each layer's parameter count. Adding layer 1, layer 2, and all the remaining layers gives 6 × layer 1 parameters + 6 × layer 2 parameters + … = 6 × total model parameters = 6N. Every layer's work is therefore included. N already accounts for the number of layers, so multiplying by 100 layers again would count the work twice.

One floating-point addition or multiplication counts as 1 FLOP. Multiplying two numbers and adding the result to a running total counts as 2 FLOPs. A GPU can execute the multiply and add as one fused instruction, but we still count two operations. A single training step contains a large number of these operations.

Start with how the model calculates a result. Multiplying inputs by their weights and adding them together is one of the most common computations in a neural network. Suppose the three inputs are 2, 4, and 6, with weights of 0.2, 0.3, and 0.5. The result is:

2 × 0.2 + 4 × 0.3 + 6 × 0.5 = 0.4 + 1.2 + 3 = 4.6

This takes three multiplications and two additions, or five floating-point operations. Extend the sum to thousands of terms, and the counts of multiplications and additions become almost equal. For large computations, we therefore usually count one multiply and one add per term, or two operations, ignoring the one fewer addition at the start of each sum.

Weight: A type of model parameter that controls how strongly an input affects a result.

Transformer: A common language-model structure in which each layer connects information from the context and then processes that representation. The architecture article will explain the inside of a layer.

Attention: The computation that selects which information in the context is useful for the current content.

Arrange many of these weighted sums together, and we have a matrix multiplication. Much of a dense Transformer's computation happens in these matrices. Adding up the work in the main weight matrices gives about 2N operations per token for the forward pass. Forward means calculating from the input through to the prediction. In practice, the GPU groups many tokens into matrices and processes them together.

Training also needs to determine where the prediction went wrong and how the parameters should change. Tracing backward from the result is called backpropagation. For the main linear computations, it requires two more kinds of matrix multiplication.

One calculates weight gradients: how changing each weight slightly affects the final error. The other calculates input gradients, passing the error signal to earlier layers. The main computation in each is comparable to the forward pass, at about 2N. Together, the backward pass takes about 4N. NVIDIA's guide to linear layers lists these three computations separately.

The main training computation for one token is therefore:

forward 2N + backward 4N ≈ 6N

Processing D tokens in total gives total compute ≈ 6ND. This basic estimate has been used for years. The compute appendix of the 2022 Chinchilla paper presents both a more detailed breakdown and the 6ND approximation. It estimates the dominant matrix computations in a dense Transformer, without requiring every small operation to follow the same ratio exactly.

The figure first adds the parameters across layers to get N, then adds the forward and backward computation per token, and finally multiplies by the total token count D. The 100 layers with 2B parameters each are only an illustration; layers can differ in size. The formula approximates the main matrix operations.

Substitute this article's model and data volume, starting with one token:

6 × 200,000,000,000 = 1,200,000,000,000 floating-point operations

That is about 1.2 trillion operations per token on average. Multiply by the total of four trillion tokens:

1,200,000,000,000 × 4,000,000,000,000 = 4.8 × 10²⁴ floating-point operations

This explains the original expression, 6 × 200,000,000,000 × 4,000,000,000,000. The right-hand side uses scientific notation to shorten a long string of zeros. Multiplying 1.2 × 10¹² by 4 × 10¹² multiplies the leading numbers to give 4.8 and adds the exponents to give 24. The result is 4.8 × 10²⁴.

Actual training does more work. Attention, for example, computes relationships among tokens, and this additional computation grows with text length. Repeated computation to save memory is also outside this basic formula. The “effective speed” used below consistently means the main workload counted as 6ND divided by actual training time. Time spent on extra computation, communication, and waiting therefore appears as a lower effective speed. We must not deduct the same overhead again.

Next, consider what each GPU can complete. Lenovo's B200 specifications updated in 2026 give a theoretical dense-compute peak of 2.25 quadrillion operations per second, or 2.25 PFLOP/s, using the 16-bit format called BF16.

Actual training must also move data, exchange results, and sometimes repeat computation to save memory. We will initially plan for an effective model-compute rate of about 44% of that theoretical peak: one quadrillion operations per second per GPU, written as 1 × 10¹⁵, or 1 PFLOP/s. The plan starts with this rate, then updates it with measurements of the B200 running the target 200B model.

Now let us calculate with 2,048 GPUs. Why start there? We want the main training run to return a result in about one month, so the team can complete an evaluation and make further adjustments. Around two thousand GPUs is a scale to examine, and 2,048 GPUs form 256 eight-GPU servers. It starts as a candidate. The calculation will show whether it fits that pace.

First, add up the speed of all the GPUs:

2,048 × 1 × 10¹⁵ = 2.048 × 10¹⁸ operations/second

Second, divide the total work by the combined speed to get the required running time in seconds:

(4.8 × 10²⁴) ÷ (2.048 × 10¹⁸) = 2,343,750 seconds

Third, one day has 24 × 60 × 60 = 86,400 seconds, so:

2,343,750 ÷ 86,400 = 27.1267 days

That is the time needed to advance training continuously. Saving progress, periodic evaluation, and failure recovery also take calendar time. We will reserve 10% of the main training window for these pauses, meaning that we expect training to advance during 90% of the window:

27.1267 ÷ 0.90 = 30.1408 days

The result under these assumptions is about 30.1 days. We will initially schedule 31 calendar days for the main training window. Procurement, installation, data preparation, preliminary experiments, and post-training need their own places in the project schedule.

In this time calculation, communication and waiting within training steps are already included in the effective rate of 1 PFLOP/s per GPU. The extra 10% covers pauses outside those steps. If we directly measure how many tokens a full week actually advances, we can use that week's average speed and avoid deducting the same waiting time twice.

BF16: A format that stores a number in 16 bits, reducing storage and computation needs during training.

FLOP: One floating-point operation, used to count computation work.

FLOP/s: Floating-point operations per second, used to measure computation speed.

PFLOP/s: 10¹⁵ floating-point operations per second, or one quadrillion.

Theoretical peak: The highest computation rate in the hardware specification.

Effective speed: The model’s main computation divided by actual training time, giving an average computation rate.

What Changes Between One Thousand, Two Thousand, and Four Thousand GPUs?

Change just one number in that calculation, and we get the other configurations. If each GPU maintains the same speed, halving the GPU count halves the combined speed, so completing the same work takes twice as long. With 2,048 GPUs, the run takes about 30.1 days. With 1,024, it takes about 60.3 days. Halve the count again to 512, and it takes about 120.6 days.

This table uses a dense 200B model, 4T tokens, an effective rate of 1 PFLOP/s per GPU, and a 90% training time fraction throughout. It shows the calculated results under these planning conditions.

For this company, 512 GPUs mean waiting four months for one main training run, which makes the feedback cycle too long. With 1,024 GPUs, that falls to two months. With 2,048, it falls to about one month, matching the pace we chose for the first plan. Increasing the count to 4,096 could theoretically save another half-month, but requires twice as many servers and supporting resources at once. We will therefore choose 2,048 GPUs for this round.

In practice, scaling also depends on how well the machines work together. As the GPU count grows, data exchange and waiting can increase, reducing the effective speed of each GPU. The table's assumption that doubling the GPUs halves the time needs to be tested at different scales.

We can also calculate how strongly speed affects the result. With the same 2,048 GPUs, if each delivers only 0.6 PFLOP/s, the time becomes 30.1408 ÷ 0.6 ≈ 50.2 days. At 1.4, it becomes 30.1408 ÷ 1.4 ≈ 21.5 days. Increasing the data volume from 4T to 8T doubles the work and takes about 60.3 days at the original speed.

The engineering team's validation task is therefore specific: first get the software working, then measure speed using the target 200B model, planned text length, and training settings. At the same time, check whether memory is sufficient, how speed changes when machines are added, and whether training can be saved and resumed correctly. Those measurements will tell us whether the 31-day plan holds up.

Cluster: A group of servers working together.

Main training window: The calendar time reserved for this large learning run, including pauses to save, evaluate, and recover. Procurement, installation, and data preparation have separate schedules.

Plan for 2,048 GPUs and About $101 Million to $109 Million in Servers

With 2,048 GPUs, the purchasing unit is 2,048 ÷ 8 = 256 eight-GPU B200 servers.

This calculation uses two public Exxact server configurations. The first uses AMD processors and 1,536GB of system memory, with eight B200 GPUs included, at a listed price of $393,607.50. The second uses Intel processors and 2,048GB of system memory, at a listed price of $423,857.50. The supporting configurations differ, and we use them to establish a current price range for the server purchase.

Using the first configuration:

256 × $393,607.50 = $100,763,520

Using the second configuration:

256 × $423,857.50 = $108,507,520

The server-purchase subtotal for this round is therefore about $101 million to $109 million. That buys 256 servers with their GPUs included. We do not need to add the price of 2,048 GPUs again.

High-speed network adapters have not been selected in these public configurations. Cluster switches, cables, shared storage, the data center, electricity, and the team also need separate calculations. Taxes, shipping, installation, and terms for volume delivery must be included in the formal quotation. Spare machines and resources for parallel experiments are separate items too. These items will increase the funding requirement, so the amount here is the server-purchase subtotal.

That gives us a concrete first plan: 2,048 B200 GPUs in 256 eight-GPU servers. With a dense 200B model, 4T tokens, an effective rate of 1 PFLOP/s per GPU, and a 90% training time fraction, the main run takes about 30.1 days, for which we schedule 31 days. Initially, allow about $101 million to $109 million for the servers.

This configuration gives the team a target it can act on. The performance validation and formal quotations can now focus on these 2,048 GPUs. Whenever measured speed, the data plan, or delivery terms change, substitute them into the calculation above and update the GPU count, schedule, and funding together.

System memory: Memory used by the server’s CPU. The 1,536GB and 2,048GB figures in these quotations refer to system memory.

GPU memory: Memory on the GPU itself. It is already included in the eight B200 GPUs in these configurations.