[Training a 500B LLM from Scratch] 1/25: Which GPUs Should We Buy?

When I started planning a 500B language model of our own, I opened the GPU product pages first. The budget was still undecided, and the purchase order was blank. Before this project could take shape, one question needed an answer: what kind of machine could make 500 billion parameters learn?

The price gap on the shelf immediately caught my attention: an RTX 5090 had an official starting price of $1,999, while the public base configuration of an eight-GPU B200 server cost more than $440,000.

Both were GPUs. One sold for thousands of dollars, the other in a system costing hundreds of thousands. What was the cheaper 5090 missing? Could a room full of them do the job?

I set the prices aside and started with something easier to check: would the model fit?

GPU: A processor suited to many calculations in parallel. It performs the main numerical computation when training a model.

Parameter: A number the model adjusts during training. 500B means 500 billion parameters.

The Cheaper Card Runs Out of Room

The RTX 5090 can train models too: its official starting price is $1,999, with 32GB of memory, 1.792TB/s of memory bandwidth, and a 575W power rating. One to eight cards can be useful for learning a training framework, building prototypes, validating multi-GPU code, or fine-tuning. Buying a few to get the code running looked attractive.

GPU memory: Memory on the GPU that holds the model and intermediate results. Its capacity limits how much one GPU can hold.

Memory bandwidth: How much data the GPU can read from or write to its memory each second.

TB/s: A trillion bytes per second, a unit of memory bandwidth used here.

GB/s: A billion bytes per second, a smaller unit of bandwidth.

But the target on the page was 500B. Putting that number into the calculation changed the picture.

32GB can hold quite a lot in everyday use. This time, it had to accommodate 500 billion parameters. Even storing the numbers needed a capacity calculation.

A 500B model has 500 billion parameters. Each 16-bit weight occupies two bytes. Multiplying 500 billion by two gives one trillion bytes, about 1,000GB, or 1TB. Storing that one copy alone needs at least 32 32GB 5090s in theory.

32 cards, roughly four eight-GPU machines. The purchase list still seemed manageable. Following the training process further, though, revealed what was missing: the model was in memory, but the records needed for learning were still outside.

After each batch of material, the training program adjusts the parameters. It needs to remember how each should change, along with the history accumulated by the optimizer. Weights, gradients, and optimizer state alone commonly add up to about 8TB, the raw capacity of 250 5090s. Intermediate layer results, communication buffers, and redundancy still have to fit. Those 250 cards are only a capacity conversion; the purchasing quantity needs more work. But the task has already grown from buying a few cards into making a large group of machines cooperate.

The 8TB estimate comes from one common arrangement using about 16 bytes per parameter: two bytes for the working weight, two for its gradient, four for a more precise weight copy used during updates, and two four-byte update records. That gives 2 + 2 + 4 + 4 + 4 = 16 bytes, so 500B × 16 bytes = 8TB. Once we choose the training software, we recalculate using its actual formats. Part Fifteen uses four-byte gradients, bringing the same calculation to 18 bytes per parameter. The scale is the point here: learning requires much more storage than one copy of the weights.

Byte: A unit of storage equal to eight bits. Capacity conversions in this article use decimal units.

Weight: A model parameter that determines how an input affects a result.

Gradient: A quantity showing how a small parameter change would affect training error. Training uses it to find an update direction.

Optimizer state: Records that the program updating model parameters keeps for later updates, such as accumulated information about earlier update directions.

The capacity calculation brought another set of costs into view: where would the extra cards go, how would they exchange results, and who would keep them running?

Installing cards together does not turn their memory into one large pool. Two 32GB cards still each manage their own 32GB. Training software has to distribute the model and join the results from different places. I followed the route that data would take.

A card can finish its own work and still have to wait for a result from somewhere else. That wait happens repeatedly during training. Impressive single-card specifications will not prevent hundreds of cards from slowing down while they wait for one another.

The 5090 lacks the high-speed connections used among GPUs inside a data center training system, so this data has to travel through server connections and the network. A 5090 is like a fast sports car. A few of them together are still fast. Once the number reaches the hundreds, the project needs a freight train whose cars are tightly connected and can be scheduled as one system.

Following the waiting problem led me to warranty and maintenance terms. Training could run continuously for weeks or months. Whenever a card failed, someone would have to locate the fault, preserve progress, replace the machine, and reconnect the job.

An intermittent error can be harder to locate than a complete failure. Time spent investigating, and how far training has to roll back, both enter the cost. The 5090 memory chips themselves have error-correction capabilities. The larger difference is the complete support system: continuous failure data, validated servers, spare parts, and field support. NVIDIA’s Founders Edition warranty also explicitly excludes data center and commercial GPU cluster use.

The cheaper line on the purchase order was starting to produce costs elsewhere: more servers, a larger network, power and cooling, recovery work, and more time waiting for results.

I did not jump straight to the most expensive machine. The professional-card page was worth a look first. Would more memory and better server support be enough?

The RTX PRO 6000 Server Edition has 96GB of error-correcting memory, about 1.6TB/s of bandwidth, and a power rating of up to 600W. It is also available in certified two, four, and eight-GPU servers. The theoretical minimum for storing 1,000GB of weights falls from 32 cards to 11. It is well suited to enterprise inference, fine-tuning, visual computing, development clusters, and medium-scale training.

It still connects multiple GPUs through the server’s general-purpose channels. Its memory bandwidth is about 1.6TB/s, while the B200 reaches as much as 8TB/s, a difference of nearly five times. Its memory is 96GB, while the B200 has 180GB. At hundreds of cards, a 500B training run pays the cost of communication and waiting again and again.

Inference and fine-tuning might accept that trade. The 500B pretraining job on the page would repeatedly move large amounts of data through those connections for a long time. Connection capacity saved today would become waiting time again tomorrow.

The 5090 and RTX PRO 6000 now had a place in the plan: development and validation. For the main pretraining run, the search would continue among data center systems.

Before putting consumer and professional cards back on the development list, I also checked AMD’s shelf. The Radeon RX 9070 XT has an official suggested price of $599, 16GB of memory, 640GB/s of bandwidth, and a 304W power rating. The Radeon AI PRO R9700 has an official suggested price of $1,299, 32GB of memory, 640GB/s of bandwidth, a 300W power rating, and error-correction support on Linux. They are well suited to low-cost development and can show the team early how much performance its code retains outside NVIDIA’s software. At 500B scale, capacity, communication among cards, and long-term operation remain the same questions.

After that pass through the catalog, cheaper cards were still worth buying. Their purpose was clearer: get small-scale development moving, then find a complete system that could cooperate reliably through the main run.

Pretraining: Training a base model on large amounts of material to develop general capabilities.

Fine-tuning: Further training of an existing model for particular tasks or data.

Inference: Using a trained model to process input and produce results.

On the Server Shelf, Why Do Eight Cards Keep Appearing?

Once I started comparing data center systems, “eight-GPU server” appeared everywhere. A moment ago I had been counting individual cards. Why had the purchasing unit become eight at a time?

The server layout made the answer clearer. One machine has to handle power, cooling, and connections together. Eight is a common standard product that vendors arrived at by solving those problems as a system.

Cards inside the same machine communicate quickly. Once data leaves the machine, it has to travel through network adapters and the cluster network, which is usually slower. That is why the eight-GPU server became a common building block.

Putting in more cards quickly makes power delivery and cooling harder. Putting in too few wastes chassis, network, and maintenance costs. Eight cards are an engineering balance that vendors developed over many years, and they later became a familiar purchasing unit.

In procurement language, one training server like this is called a node. Many connected nodes form a training cluster. We can add nodes one at a time, or buy a vendor’s integrated 72-GPU rack. The latter puts compute, networking, and liquid cooling in one rack, while its price and facility requirements rise with it.

The purchase sheet changed accordingly. Next to the price of one GPU, it now needed a server price, an inter-server network, and a way to expand. The working unit had become the whole cluster.

This decision would stay with the project: how to add machines over the next three years, how to repair failures, and which platform the training software would grow around.

Node: One server in a cluster. This article commonly uses a server containing eight GPUs as one node.

Cluster: Multiple servers connected by a network and working together.

Inside the Data Center, GPUs Are Not the Only Option

Looking again with system and operating costs in mind left fewer candidates and a more concrete question: on the day training begins, which system can be delivered, run the code, and expand afterward?

The H200 is the mature choice. Each card has 141GB of memory, 4.8TB/s of bandwidth, and a power rating of up to 700W. There is already extensive software and operating experience. Its value is that it can get the project started sooner.

If the company must begin training within a few months, maturity itself is performance. Saving one month on software work, waiting for machines, or changing the facility may matter more than making each card slightly faster.

The B200 increases memory to 180GB per card and bandwidth to as much as 8TB/s. Eight cards in one machine also have a dedicated high-speed connection. On October 27, 2026, the Exxact eight-GPU AMD configuration had a base price of $443,278, or $457,798 after selecting eight training NICs. It is more expensive, each card draws up to 1,000W, and the facility requirements are higher. In return, it offers less waiting and faster training.

This public configuration gave me a starting point for the rest of the calculations. I marked B200 as the reference: work through capacity, time, and cost, then compare other routes against the same job. The delivery date for the full quantity would still need confirmation during procurement.

AMD’s MI355X has 288GB of memory, 8TB/s of bandwidth, and a power rating of up to 1,400W. On paper, it is very attractive. It gives us another route that can genuinely compete, while requiring the team to validate AMD’s software, communication among cards, and cluster stability with its own code.

The larger memory made it worth investigating, but software migration would take time too. Three items went on the test list: sustained training, the speed lost as more machines join, and time to repair a problem. Cheaper hardware would have to pass those tests to save money overall.

If the project does not begin training until 2027, the candidate list will change again. B300 and Rubin eight-GPU systems continue to increase memory and bandwidth. B300 has 288GB per card, while Rubin reaches up to 288GB and 22TB/s per card. The MI455X reaches 432GB and 23.3TB/s, and is delivered mainly as part of the 72-GPU Helios rack. They are likely to be faster, but complete prices, power requirements, and delivery schedules still need formal quotations.

Newer specifications were tempting, but the training calendar would keep moving. I added the start date to the comparison: systems available now and systems for 2027 had to be evaluated with the cost of waiting included.

After looking at next-generation racks, one question remained: did we have to buy GPUs and bring them back to a facility to train this model?

Google TPU is a parallel route. Ironwood TPU7x became generally available on March 31, 2026, with 192GiB of memory per chip. Google’s documentation explicitly supports large-model pretraining from scratch, as well as JAX and PyTorch. TPU8t and TPU8i are currently marked as coming soon, so the current evaluation should use TPU7x.

The transaction model also changes with TPU. A company usually rents compute in Google Cloud or signs a long-term capacity agreement, instead of buying individual chips and moving them into its own facility. This can remove some of the work involved in building the hardware system. It also creates new questions: how long the existing training code will take to migrate, whether enough cluster capacity will be available when training begins, and how a long-term capacity agreement will be priced. TPU7x should enter the main candidate set alongside NVIDIA and AMD. Costs should be compared based on completing the same training job. A cloud rental rate and the purchase price of a server cannot be subtracted directly.

Cerebras is another design. It puts a large amount of compute on one very large wafer-scale chip, while the surrounding system keeps supplying it with data. Its pretraining documentation describes a complete training path. Customers can buy the whole system or use dedicated cloud training. This differs from buying GPU servers and from calling a public inference API.

This route has a real training record. In a 2026 review, Cerebras wrote that Jais 2 went through end-to-end training and its 8B and 70B variants were released in December 2025. However, the CS-4, announced in August 2026, was presented mainly around inference speed and initial shipment plans. It cannot be treated directly as evidence of 500B pretraining performance. To put Cerebras into the formal competition, the vendor needs to prove the target model’s training path and provide a complete price, delivery schedule, and expected training time.

To stop circling through model numbers, I put the six routes on one map: consumer GPUs, professional GPUs, NVIDIA data center GPUs, AMD data center GPUs, Google TPU, and Cerebras. The comparison focuses on three things: how each is acquired, its strengths, and its main constraint.

With the candidates assembled, I looked for actual training records. Public reports from frontier-model companies did not give the table to a single vendor. Some deployed several routes; others assigned different jobs to different chips.

The same applies to next-generation systems. Cognition already runs a production workload on a Vera Rubin 72-GPU rack, which shows that the system has entered production. Training a 500B model from scratch still needs separate validation.

Those records strengthened the case for the candidates and brought the question back to our workload. A system that worked well elsewhere was useful evidence. Choosing for this 500B plan meant comparing them with the same training code.

TPU: Google’s processor designed for machine learning. This article compares training capacity obtained through Google Cloud.

Training framework: Software that organizes model computation and parameter updates. PyTorch and JAX are examples used here.

Wafer-scale chip: A chip design that places a large computing system across a whole silicon wafer. Cerebras follows this route.

Choose a Ruler First

When I closed the product pages, the quantity on the order was still blank, but the selection criteria had changed. The comparison was about the time and resources needed to complete the same training run. What was still missing was how much work that run contained.

B200 is shipping, with public server configurations and prices, and represents a Blackwell system that can be deployed today. I decided to use it to connect capacity, time, and budget first. Other candidates would bring their results to that comparison, and the final purchase would follow testing and delivery conditions.

Before a formal order, H200, B300, MI355X, and TPU7x must all run the same reduced model. This test can only verify that the software runs, measure single-system efficiency, and reveal early stability. Its result cannot be scaled directly into a 500B training speed. The next step is to use multiple machines to test communication, failure recovery, and the speed after more machines are added, and to confirm the result with the target 500B workload. Cerebras also needs the vendor to complete the same validation. If the project begins in 2027, Rubin and MI455X will also enter the main candidate set.

Owned servers, cloud compute, and complete proprietary systems present prices in different forms. In the end, all of them must be converted into the time, power, engineering effort, and three-year total cost required to complete the target training run. Testing and quotations will gradually complete those accounts. First comes the immediate question: how many B200s would this 500B model need to finish training in an acceptable time?