[Training a 500B LLM from Scratch] 25/25: Once the Model Works, How Does the Investment Keep Creating Value?

The chat window could now serve users. I opened the service records beside the original purchasing sheet. On the left, servers and switches added up to more than three hundred million dollars. On the right, a serving-cost calculation put one request at a few cents.

How could the same machines appear in two figures so far apart? What did those few cents contain, and where did the original investment belong?

I set the calculator aside and reorganized the records around what the machines did: how much we paid to acquire them, how long a model version occupied them, and how many requests they completed after launch. Once each of those accounts answered its own question, we could decide where the next round of money should go.

Bring Back the Ledger from Part 3

First, I returned to the purchasing list in Part 3. There were 640 main training servers and 20 spares: 660 servers and 5,280 B200 GPUs in total, with 5,120 participating in main training. The machines now serving users came from the same pool.

At the public configuration prices recorded on October 27, 2026, each of the 660 servers, including eight specified network adapters, cost $457,798, for a total of $302,146,680. The 416 specified switches cost $34,555 each, totaling $14,374,880. Together, the two items came to $316,521,560.

I kept the quote date and configuration beside each amount. Procurement would use the same list to obtain volume prices, delivery dates, and payment terms, updating the record as each arrived. The network adapters were already included in the server price and could not be charged again in the network budget.

The first-year personnel cash budget was $5,822,096. Electricity, calculated from the planned operating load and electricity-price scenario, was $8,176,000. Nine months of planned facility rent adds $20,047,500, bringing the current planning subtotal to $350,567,156.

Blank rows remained below: transceivers and fiber matched to distance, shared storage and the management network, shipping, installation, taxes, data, and external services. Part 3 planned to lease 13.5MW of capacity for nine months starting in month 4. As of November 4, 2026, it uses a planning rent of $165 per kW per month, excluding electricity, or $2,227,500 per month and $20,047,500 for nine months, until the facility contract provides the supplier price.

This sheet first answered how much cash the company needed to prepare. The current planning subtotal of about $350.57 million had a traceable basis. Contracts would fill the remaining lines, and a 10% contingency would be added to the complete budget. Yet even a fully populated sheet would leave another question unanswered: how much of that investment had the model version we just trained actually used?

Procurement cash: Money actually paid for equipment under the purchase contract. Payments may occur in stages, while the purchasing list still records the complete equipment configuration.

Buying the Machines and Training One Model Are Different Accounts

The service window was still running, and the next evaluation jobs were queued. After training one version, the same server could continue with research, material generation, or user requests. The purchase paid for those later uses too, so the whole amount could not fall on the first main training run. Nor could the investment in the machines disappear from view, leaving only the monthly electricity bill.

Beside the cash ledger, I added a resource ledger. The first tracked actual monthly payments and whether funding was sufficient. The second tracked GPU hours, storage, people, and services by job, showing what each version used. Serving would then calculate cost per request from its own consumption.

The same equipment can be viewed in three ways. The cash ledger records payments; the resource ledger allocates usage to training, fine-tuning, and serving; the serving ledger calculates cost per request. Allocation connects them. Their totals must not be added together again. The colored bars in the middle indicate categories of work, with lengths shown only for illustration.

Main training went in first. The original schedule of 76 days on 5,120 active GPUs corresponded to:

5,120 × 76 × 24 = 9,338,880 GPU hours

Part 2 calculated a theoretical duration of about 75.35 days; 76 days was the scheduling allowance. With job records available, we could replace the plan with actual consumption item by item. Scheduler records supplied machine time, training logs supplied effective tokens, and separate columns held failures, saving and recovery, and waiting. Putting planned and actual use together let us investigate the difference.

After filling the main training row, the job directory still contained earlier small experiments and failed attempts, followed by SFT, reward training, and acceptance testing. They had consumed the same resources. I assigned each job ID to its stage. Only after adding them together could we see the full path to a releasable version.

GPU hours also needed a consistent internal price. Finance and I started with a simplified comparison that spread server capital cost over three years with zero residual value, then added network and facility allocations. Every task would use the same basis. Formal accounting depreciation and tax treatment would follow company policy; this internal resource sheet would help engineering compare its next choices.

GPU hour: A unit of resource use equal to one GPU occupied by a task for one hour.

Cost allocation: Assigning shared equipment and service costs to the tasks that use them under a consistent set of rules.

Why Do the Same Machines Cost More When We Use Them Less?

While pricing GPU hours, another gap appeared. A machine could spend a whole year in the facility without having worthwhile work for every hour. Idle time produced nothing, while the purchase remained on the books.

I started with the server alone. An eight-GPU machine priced at $457,798, available continuously for three years of 365 days each, offered 210,240 GPU hours. Dividing the purchase price by that capacity gave about $2.18 per available GPU hour.

Replacing available time with time assigned to useful work changed the result. At 70% utilization, the server capital component was about $3.11 per effective GPU hour. At 35%, it rose to about $6.22. Electricity, networking, facilities, and operations still had to be added. The machine’s price had not changed, but its capital cost per unit of output had doubled.

That difference brought me back to the resource plan after main training. We would first reserve capacity for live serving and failure replacement, then allocate evaluation, teacher generation, and later research according to need. Each pool would record both completed work and queues. Capacity that remained idle would be reassigned, and the next purchasing plan adjusted accordingly.

The decision to scale the next model would go through the same sheet. How many more GPU hours would a larger model need, and which capability would it improve? We would first measure the benefit on comparable tasks. When a main training run ended, the machines would have a defined next job, and new investment would have a basis for comparison.

Utilization: The share of resources actually used within a chosen time period and scope. Here we examine both task occupancy and useful output.

What Does One Request Cost?

With resources reassigned, I returned to the few-cent serving calculation on the right. Part 24 measured how much traffic one replica could handle within the latency target. Now we needed to include the cost of keeping that replica running.

Let C be all serving expenditure for a month and N the number of completed requests. Cost per request is C ÷ N. C includes GPU allocation, CPU and memory, networking and storage, facility power, and direct operations. Earlier training and research investment belongs in a separate recovery account, giving operating expense and model development cost their own places.

In the cost sheet, I first selected $4 per GPU hour to examine one 16-GPU replica. Running around the clock for a 730-hour month gave a GPU-related cost of $46,720. At an average of 1 completed request per second, monthly volume was about 2,628,000 requests, or roughly $0.0178 per request for the GPU portion. At 0.1 requests per second, the same expense became about $0.178 per request.

The $4 cell was a sensitivity-analysis input, to be replaced with formal quotes and internal resource costs. The completion rate was averaged over the entire billed month, so quiet periods were already included. Multiplying by utilization again would count them twice.

The few cents finally had a numerator and denominator, but a new question lay inside the requests themselves. Pasting an entire manual and asking one short question consumed very different resources. I recorded input length, output length, cache hits, and latency tier separately, then calculated each group’s cost so that an average would not hide the expense of long documents.

Unit cost: The cost of completing one request, generating a given number of tokens, or solving one task. The measurement scope must be stated alongside it.

A Cheap Answer Still Has to Finish the Job

After grouping requests, the smaller model’s low price stood out. One option cost $0.002 per call with a 50% task success rate; another cost $0.01 with a 90% success rate. The cheaper call might require a retry, a human correction, or eventual escalation to the larger model. The user’s task did not automatically end when the first answer appeared.

I followed each request through to the end of the task and added “total cost per successful task.” Attempts and review costs for the same task were recorded together. Success followed the criteria from Part 23, and failures and human handoffs remained in the record. Only then could we see where a lower price actually saved money.

The task records suggested a division of work. Clear, short tasks could go to a smaller model first; complex long documents and difficult problems could go to the 500B model. When new facts were needed, retrieval would come first, followed by an answer grounded in the material. The router deciding where to send requests would also be evaluated, especially for difficult tasks mistakenly left with a model that could not handle them.

The first version would route using readily identifiable task categories and input features, with a small random sample checked manually. Once the records stabilized, we could train a more detailed routing model. The required quality and privacy boundaries would stay consistent. What changed was who performed the work within those boundaries.

By 2026, public inference frameworks offered mechanisms such as caching, chunked input processing, and scaling across replicas. These changed serving efficiency; we would still evaluate the final output with the same tasks. vLLM’s optimization documentation could provide a starting point for serving experiments, while the project’s logs would determine the actual savings.

How Do New User Problems Return to Training?

As task records accumulated, failures began to take different forms. Some questions needed recent facts. In others, the model already had the right material but cited it incorrectly. Sending all of them straight into retraining would quickly enlarge the next GPU bill. I first traced each gap to the part of the system where it arose.

Missing new facts called for better retrieval and updated sources. Incorrect citations from material already in the input called for a review of prompts and reading ability. Changed tool interfaces called for adapter repairs and tests. Repeated misunderstanding of the same kind of request would justify new supervised demonstrations; tasks that required learning through attempts would need verifiable environments and reward training. The repair followed the problem, with training as one step serving a specific purpose.

Material that genuinely belonged in the curriculum returned to the pipeline from Parts 7 through 10: record its source and conditions of use, clean it, deduplicate it, review it, and fix a version. User content would be handled within its authorized scope. Failure cases used for improvement would enter the regression set, while test separation remained in place and fresh questions were added for acceptance.

For this round, I narrowed the target to incorrect citations in long documents while preserving math and coding performance. Another candidate tried shorter answers, with completion rates checked alongside length. Each batch would compare a few clear changes, move through offline evaluation, a limited rollout, and a cost review, then earn a decision on wider use.

The records gradually established a rhythm: review problems and operating costs every week, and compare a batch of small experiments every two weeks. Important failures would be handled immediately. Ordinary improvements would accumulate enough evidence before training, and acceptance would determine release. The cluster’s next stretch of time would be scheduled around these concrete, smaller questions.

Open the Original Purchase Order Again

At the beginning, the quantity column on the purchase order was empty. Beside it now sat 10T cumulative input tokens, 5,120 active B200 GPUs, a 76-day main training schedule, and a purchase of 660 servers. Further along were model versions, user requests, and the record of each improvement.

Each number had a place from which it could be updated. Contracts would complete the purchasing inputs. Measured throughput would revise the schedule. Evaluation reports would check capability. Production logs would supply serving cost. We would keep the gaps between plan and reality too, so the next round could show where our earlier judgment had missed.

The two accounts on the desk finally connected. The cash sheet would keep filling in the remaining quotes. Research would record GPU hours by task. Live serving would track quality, latency, and unit cost together. Tasks users could not complete would, after review, become starting points for new material, product fixes, or training experiments.

When we began buying GPUs, the question was what machines would let this model learn. Now the machines had continuing work, and the question had become where the next unit of compute should go. Which capability deserved more training? Which task could be handled well through retrieval, a smaller model, or a product change? The answers would live in these records. Owning a model had finally grown from a purchase order into a way of continuing to make decisions and improve.