[Training a 200B LLM from Scratch] 1/25: Which GPUs Should We Buy?

Imagine a company preparing to invest a large amount of money in the United States to train its own 200B model.

The first decision in a project like this is often reduced to “find the fastest GPU and buy a lot of them.” Once you open the catalog, however, two very different products sit in front of you: an RTX 5090 with an official starting price of $1,999, and an eight-GPU B200 server with a public price close to $400,000.

Some people will ask: why not buy a room full of 5090s?

That question deserves a serious calculation. If the only explanation is “data center GPUs are more professional,” then the price difference of dozens of times remains unexplained. We need to break down, item by item, what that money actually buys.

GPU: A processor suited to many calculations in parallel. It performs the main numerical computation when training a model.

Parameter: A number the model adjusts during training. 200B means 200 billion parameters.

The Answer First: The 5090 Can Train, but the Economics Change at Scale

The RTX 5090 can certainly train models. Its official starting price is $1,999. Each card has 32GB of memory, 1.792TB/s of memory bandwidth, and a 575W power rating. One to eight cards are well suited to learning a training framework, building prototypes, validating multi-GPU code, and fine-tuning models. At the beginning of a project, a team can absolutely use them to get the code running.

GPU memory: Memory on the GPU that holds the model and intermediate results. Its capacity limits how much one GPU can hold.

Memory bandwidth: How much data the GPU can read from or write to its memory each second.

TB/s: A trillion bytes per second, a unit of memory bandwidth used here.

GB/s: A billion bytes per second, a smaller unit of bandwidth.

The problem is the number 200B.

One card looks large because the models we usually run are much smaller than 200B. Once the model grows to 200 billion parameters, 32GB suddenly changes from “a lot” into a very small box.

A 200B model has 200 billion parameters. Storing one copy of the 16-bit weights commonly used during training already takes about 400GB. Before any computation begins, the theoretical minimum is at least thirteen 32GB 5090s.

Thirteen cards do not sound frightening. It is easy to think that two machines would solve the problem. But this only stores the model weights. It has not allowed the model to learn anything yet.

Those thirteen cards only hold one copy of the weights. Training also needs to remember how each parameter should change and the state the optimizer has accumulated from earlier updates. Counting only weights, gradients, and optimizer state, a common estimate is already about 3.2TB, equal to the raw capacity of one hundred 5090s. This still excludes temporary results from each layer, communication buffers, and redundancy. One hundred is not a purchasing estimate. It simply shows that training a 200B model moves the project from buying a few cards to building a cluster.

Weight: A model parameter that determines how an input affects a result.

Gradient: A quantity showing how a small parameter change would affect training error. Training uses it to find an update direction.

Optimizer state: Records that the program updating model parameters keeps for later updates, such as accumulated information about earlier update directions.

At this point, the price question becomes: how many machines, how much networking, and how many people are needed to look after each kind of cluster?

GPU memory does not merge automatically. Two 32GB cards remain two separate 32GB spaces. The training software has to split the model and make the cards exchange results continuously. The more cards there are, the more data has to move among them.

Every training step works this way: each card computes its own part, then waits for the other cards to send their results. No matter how fast the computation is, the whole cluster cannot be fast if it spends too much time waiting.

The 5090 lacks the high-speed connections used among GPUs inside a data center training system, so this data has to travel through server connections and the network. A 5090 is like a fast sports car. A few of them together are still fast. Once the number reaches the hundreds, the project needs a freight train whose cars are tightly connected and can be scheduled as one system.

Then there is operating time. Training may continue for weeks or even months. If one card fails, the team needs to know where the failure occurred, save progress, replace the machine, and continue.

Imagine that training has run for two weeks and one card begins reporting occasional errors. The expensive part is how long the team needs to locate the problem and how far back the entire run must restart. The 5090 memory chips themselves have error-correction capabilities. The larger difference is the complete support system: continuous failure data, validated servers, spare parts, and field support. NVIDIA’s Founders Edition warranty also explicitly excludes data center and commercial GPU cluster use.

The money saved on 5090 cards therefore moves into more servers, a larger network, harder power and cooling problems, more failure-recovery work, and a longer training time.

Continue down the aisle and professional cards offer a middle route: more like a server product than a consumer card, without jumping straight to a $400,000 machine.

What about a professional card? The RTX PRO 6000 Server Edition has 96GB of error-correcting memory, about 1.6TB/s of bandwidth, and a power rating of up to 600W. It is also available in certified two, four, and eight-GPU servers. The theoretical minimum for storing 400GB of weights falls from thirteen cards to five. It is well suited to enterprise inference, fine-tuning, visual computing, development clusters, and medium-scale training.

It still connects multiple GPUs through the server’s general-purpose channels. Its memory bandwidth is about 1.6TB/s, while the B200 reaches as much as 8TB/s, a difference of nearly five times. Its memory is 96GB, while the B200 has 180GB. At hundreds of cards, a 200B training run pays the cost of communication and waiting again and again.

If a team mainly does inference or fine-tuning, this trade may be attractive. Our task is to train a 200B model from scratch. The machines will exchange large amounts of data repeatedly for a long time, so the same gap appears every day.

The answer is therefore simple. Both the 5090 and RTX PRO 6000 can technically train models. This 200B project would use them for development and validation, while assigning the formal large-scale pretraining run to a data center system.

Beyond NVIDIA, there is AMD. The Radeon RX 9070 XT has an official suggested price of $599, 16GB of memory, 640GB/s of bandwidth, and a 304W power rating. The Radeon AI PRO R9700 has an official suggested price of $1,299, 32GB of memory, 640GB/s of bandwidth, a 300W power rating, and error-correction support on Linux. They are well suited to low-cost development and can show the team early how much performance its code retains outside NVIDIA’s software. At 200B scale, capacity, communication among cards, and long-term operation remain the same questions.

The value of these cards is also clear: they are good development tools. Development tools and formal training systems solve problems at two different stages.

Pretraining: Training a base model on large amounts of material to develop general capabilities.

Fine-tuning: Further training of an existing model for particular tasks or data.

Inference: Using a trained model to process input and produce results.

Why Everyone Talks About Eight-GPU Servers

The next question is: why always eight cards?

Eight comes from an engineering balance. Vendors turn the number of cards that one machine can power, cool, and connect well into a standard product that can be delivered repeatedly.

Cards inside the same machine communicate quickly. Once data leaves the machine, it has to travel through network adapters and the cluster network, which is usually slower. That is why the eight-GPU server became a common building block.

Putting in more cards quickly makes power delivery and cooling harder. Putting in too few wastes chassis, network, and maintenance costs. Eight cards are an engineering balance that vendors developed over many years, and they later became a familiar purchasing unit.

In procurement language, one training server like this is called a node. Many connected nodes form a training cluster. We can add nodes one at a time, or buy a vendor’s integrated 72-GPU rack. The latter puts compute, networking, and liquid cooling in one rack, while its price and facility requirements rise with it.

This is also why comparing the price of one GPU can mislead us. The actual working unit has moved from one card, to one server, and finally to the entire cluster.

So we are choosing more than a chip. We are choosing how to add machines over the next three years, how to handle failures, and which platform the software team will work around.

Node: One server in a cluster. This article commonly uses a server containing eight GPUs as one node.

Cluster: Multiple servers connected by a network and working together.

Inside the Data Center, GPUs Are Not the Only Option

The data center shelf has fewer models than the consumer market. Only a handful of routes are really worth comparing now.

The question should also be asked differently: when training is ready to begin, which system can actually be delivered, can scale, and can be used by the team?

The H200 is the mature choice. Each card has 141GB of memory, 4.8TB/s of bandwidth, and a power rating of up to 700W. There is already extensive software and operating experience. Its value is that it can get the project started sooner.

If the company must begin training within a few months, maturity itself is performance. Saving one month on software work, waiting for machines, or changing the facility may matter more than making each card slightly faster.

The B200 increases memory to 180GB per card and bandwidth to as much as 8TB/s. Eight cards in one machine also have a dedicated high-speed connection. Public eight-GPU configurations are about $393,200 to $423,900. It is more expensive, each card draws up to 1,000W, and the facility requirements are higher. In return, it offers less waiting and faster training.

The B200 is a useful center point in the current comparison. It is already available, public prices are visible, and its performance is representative of data center training systems that can be deployed today. It can serve as a baseline for capacity, time, and cost calculations.

AMD’s MI355X has 288GB of memory, 8TB/s of bandwidth, and a power rating of up to 1,400W. On paper, it is very attractive. It gives us another route that can genuinely compete, while requiring the team to validate AMD’s software, communication among cards, and cluster stability with its own code.

That validation needs a sustained run. We need to see whether training can continue, how much speed is lost as machines are added, and how long the team takes to fix a problem. A lower hardware price can still produce a higher total cost when software migration is slow.

If the project does not begin training until 2027, the candidate list will change again. B300 and Rubin eight-GPU systems continue to increase memory and bandwidth. B300 has 288GB per card, while Rubin reaches up to 288GB and 22TB/s per card. The MI455X reaches 432GB and 23.3TB/s, and is delivered mainly as part of the 72-GPU Helios rack. They are likely to be faster, but complete prices, power requirements, and delivery schedules still need formal quotations.

Waiting for the next generation also has a cost. If the model launches one year later, the money saved on hardware may be much smaller than the value of the lost time. “Start training now” and “start training in 2027” can therefore produce different answers.

If the goal is to train a model, the options do not have to stop at GPUs.

Google TPU is a parallel route. Ironwood TPU7x became generally available on March 31, 2026, with 192GiB of memory per chip. Google’s documentation explicitly supports large-model pretraining from scratch, as well as JAX and PyTorch. TPU8t and TPU8i are currently marked as coming soon, so the current evaluation should use TPU7x.

The transaction model also changes with TPU. A company usually rents compute in Google Cloud or signs a long-term capacity agreement, instead of buying individual chips and moving them into its own facility. This can remove some of the work involved in building the hardware system. It also creates new questions: how long the existing training code will take to migrate, whether enough cluster capacity will be available when training begins, and how a long-term capacity agreement will be priced. TPU7x should enter the main candidate set alongside NVIDIA and AMD. Costs should be compared based on completing the same training job. A cloud rental rate and the purchase price of a server cannot be subtracted directly.

Cerebras is another design. It puts a large amount of compute on one very large wafer-scale chip, while the surrounding system keeps supplying it with data. Its pretraining documentation describes a complete training path. Customers can buy the whole system or use dedicated cloud training. This differs from buying GPU servers and from calling a public inference API.

This route has a real training record. In a 2026 review, Cerebras wrote that Jais 2 went through end-to-end training and its 8B and 70B variants were released in December 2025. However, the CS-4, announced in August 2026, was presented mainly around inference speed and initial shipment plans. It cannot be treated directly as evidence of 200B pretraining performance. To put Cerebras into the formal competition, the vendor needs to prove the target model’s training path and provide a complete price, delivery schedule, and expected training time.

The figure below places six routes together: consumer GPUs, professional GPUs, NVIDIA data center GPUs, AMD data center GPUs, Google TPU, and Cerebras. It compares only three things: how each route is acquired, where it is strongest, and what its main constraint is.

Looking at companies that actually train frontier models, we find that they have not produced one standard answer.

Some companies use NVIDIA’s newest systems. Some deploy AMD at the same time. Some build their own chips. Even within one company, different chips may handle different jobs.

The same applies to next-generation systems. Cognition already runs a production workload on a Vera Rubin 72-GPU rack, which shows that the system has entered production. Training a 200B model from scratch still needs separate validation.

The industry therefore has no universal accelerator. Everyone chooses according to the model, software, supply relationships, and training start date. Other companies’ lists can help us narrow the range. The final decision still needs a run with our own workload.

The procurement task can therefore be reduced to one concrete exercise: narrow the candidates to a few options, then let the same training code decide.

TPU: Google’s processor designed for machine learning. This article compares training capacity obtained through Google Cloud.

Training framework: Software that organizes model computation and parameter updates. PyTorch and JAX are examples used here.

Wafer-scale chip: A chip design that places a large computing system across a whole silicon wafer. Cerebras follows this route.

Choose a Ruler First

We still do not know how many cards the final system should contain. Without the model’s compute requirement and training time, that number cannot be calculated.

Giving a card count now would look precise while hiding assumptions that have not been validated. A more reliable approach is to choose a ruler first, then calculate the numbers one by one.

The B200 is shipping, has public server configurations and prices, and its performance represents a Blackwell system that can actually be deployed today. This project will therefore use B200 to establish a reproducible reference for capacity, time, and budget. The reference only gives the comparison a common starting point. It does not assume a winner.

Before a formal order, H200, B300, MI355X, and TPU7x must all run the same reduced model. This test can only verify that the software runs, measure single-system efficiency, and reveal early stability. Its result cannot be scaled directly into a 200B training speed. The next step is to use multiple machines to test communication, failure recovery, and the speed after more machines are added, and to confirm the result with the target 200B workload. Cerebras also needs the vendor to complete the same validation. If the project begins in 2027, Rubin and MI455X will also enter the main candidate set.

Owned servers, cloud compute, and complete proprietary systems present prices in different forms. In the end, all of them must be converted into the time, power, engineering effort, and three-year total cost required to complete the target training run. Once those numbers are calculated, the data can decide which route fits best.