[Training a 500B LLM from Scratch] 24/25: How Do We Put the Model in Users’ Hands?

The last round of evaluation left C as our release candidate. I gathered its weights, tokenizer, and chat template, ready to open a chat window people could actually use.

During training, 5,120 B200 GPUs worked on the same task: read material, compute gradients, and update parameters. Now users sat on the other side of the screen. One asked a single question, another pasted in an entire manual, and a third finished reading an answer and immediately followed up. The machines had to accept these requests of different lengths and send the answers back in time.

The first step was one person and one request. Once that path worked, we could ask how many machines we needed when more people arrived.

Put One Model on Two Servers First

At BF16, each of the 500B parameters occupies 2 bytes, making the raw weights about 1TB. One server has eight B200 GPUs with 180GB each, or 1.44TB in total. Spread the weights evenly, and each GPU already holds about 125GB.

Some memory remains, but answering questions needs working space too. The long document a user pastes in, the records retained during generation, and temporary results all have to live somewhere. Fitting the weights does not yet answer what happens when dozens of people arrive together.

I started with two eight-GPU servers hosting one model replica. The first handled the earlier part of the model, and the second continued through the later part. Within each server, eight GPUs shared the computation of each layer. That gave us TP8 within a machine and PP2 across machines. Across 16 GPUs, the weights averaged about 62.5GB per GPU, leaving more room for requests.

The handoff had a cost: intermediate results had to cross between machines, potentially increasing the wait for a single request. I kept the eight-GPU option as a compact comparison, while using 16 GPUs for the first round of loading, answering, and capacity checks.

BF16: A numerical format that uses 16 bits, or 2 bytes, per value. Here it stores the weights used in computation.

Serving replica: A set of resources that can independently perform model inference. One replica can span multiple GPUs or servers.

TP8: Tensor parallelism of degree 8, distributing computation within a layer across eight GPUs.

PP2: Pipeline parallelism of degree 2, splitting the model’s layers into two stages that two groups of GPUs execute in sequence.

Fix One Serving Path First

The servers were ready, but software still had to load the weights, schedule computation, and receive requests. Between vLLM and SGLang, I chose vLLM first. Its parallelism documentation described a TP8-plus-PP2 arrangement on two eight-GPU machines, a useful starting point for this integration.

That choice let me fix one set of startup, monitoring, and load-test configurations. We could check the model format, support for the parallel split, and memory use along the same path. SGLang’s server arguments also covered parallelism, caching, and serving settings. I kept it for the next comparison under equivalent conditions.

After exporting the weights, I first supplied the same inputs used in the training environment and checked output distributions and task scores. Once conversion was correct, we looked at peak memory and sustained operation. Versions, images, templates, and startup arguments were saved together, giving every later optimization a reliable reference.

The gateway checked identity and request quotas. A router sent requests to healthy replicas, vLLM generated answers, and monitoring recorded time and errors. We assigned serving machines their own partition of resources, so they could focus on requests while users were waiting.

Once the path was connected, I pasted in a 4,000-token document and asked for a summary. The screen stayed empty for a moment. Then the text began to appear.

Router: Here, the service component that assigns requests to available model replicas.

Why the Pause Before the Words Start Appearing?

During the initial wait, the model was reading the material I had pasted in. It had to compute representations for those positions before it could begin answering. Many input positions could be processed together. This phase is called prefill.

Once the first answer token appeared, the rhythm changed. The model appended the token it had just generated, predicted the next one, and fed the new result back in. Each next step had to wait for the previous one to be determined. This gradual writing phase is called decode.

“Fast” therefore meant two things to the user: how soon the first output appeared, and how quickly the rest followed. Long documents added work at the beginning; long answers kept generation running longer. I recorded the phases separately instead of looking only at the total seconds per request.

A second question followed immediately. Did generating each new token require computing all 4,000 earlier positions again?

Prefill: The phase in which the model processes the existing input context to prepare for generation.

Decode: The phase in which the model generates new tokens one step at a time from the existing context.

The Earlier Text Has Not Changed. Keep Its Records.

Part 6 introduced Q, K, and V in Attention. As generation moves forward, the K and V for earlier positions can be reused. The engine keeps them in GPU memory, reads them directly at the next step, and adds the K and V produced for each new position.

Read the diagram from the left. Prefill processes the input, retains its K and V, and produces the first answer token. On the right, decode takes the token just generated, reuses the earlier records, predicts the next token, and appends the new position’s K and V. The two MODEL blocks show the same model operating in two phases. The small cache blocks represent groups of vectors at each position, each containing multiple values.

This is the KV cache. It saves repeated computation and consumes some of the memory we reserved earlier.

I broke the calculation down to one layer and one position. Consider a single pair of K and V vectors, each with 128 values at 2 bytes per value. That small record takes 2 × 128 × 2 = 512 bytes. The model has many layers, each retaining its own records, and a layer may contain multiple K and V pairs. More history means more records.

More importantly, the replica shares its model weights across requests, while the cache follows each request. Two people supplying different documents each create their own history. The cache usually keeps growing until an answer finishes. Afterward, the relevant space can be released, or reusable prefixes retained according to the engine’s rules.

This explains why one working request does not imply that dozens can run together. The next memory peak to measure was the one produced by the target input length, output length, and concurrency at the same time.

KV cache: A cache of K and V produced by earlier tokens in Attention, allowing later generation to reuse those results.

Concurrency: The number of requests still being processed at the same time.

From one layer and one position to a complete request

A pair of K and V vectors retained in each layer is called a KV head. The head dimension is the number of values in each vector. Conventional KV tensor storage can be calculated as:

KV bytes = 2 × layers × KV heads × head dimension × tokens × bytes per element

The initial 2 accounts for K and V. With 80 layers, 8 KV heads, 128 values per head, and 2 bytes per element, each token requires 327,680 bytes. A context of 8,192 tokens takes about 2.68GB; 32 requests of that length take about 85.9GB. As output grows, so does the required capacity.

The deployment calculation uses the actual layer count, KV head count, and dimensions in the model configuration. This formula covers conventional cache tensors. Allocators, alignment, and runtime state require additional space. Compressed caches or other Attention structures need a calculation based on their implementation.

A Second Person Arrives, and the Queue Changes

After one request worked, I changed the test to several people asking questions together. Some answers needed only a few dozen tokens; others were still producing long explanations. If we waited for the longest request in a batch to finish before accepting the next batch, the positions already freed would sit idle.

Continuous batching used those openings. Finished requests left, new requests joined, and answers still being generated kept moving. Requests of different lengths could share a computational batch. Scheduling records now mattered: we wanted the GPUs to keep working while watching how long each person actually waited.

Repeated material offered another opportunity. If users kept asking about the same manual, or every request carried the same system instructions, a matching prefix could reuse existing KV. vLLM’s prefix caching documentation explained that path. I divided the load test into distinct-input and shared-prefix groups, measured cache hits and misses separately, and planned capacity around the actual repetition rate.

Long inputs could still monopolize time and interrupt short answers already being generated. Chunked prefill split the work on a long input so it could be scheduled alongside decode. I gradually adjusted the token limit per batch, watching time to first token and the intervals between later tokens until both were acceptable.

For the first version, the same replica continued to perform prefill and decode. Deploying the phases separately would let them scale independently, but would also require KV transfers and more scheduling and network work. I first measured the current queue to find which phase was constrained, then would decide whether separating them was worthwhile.

Continuous batching: Dynamically adding and removing requests during generation so that requests of different lengths can share computational batches.

Prefix caching: Reusing the cache produced by an exactly matching input prefix to reduce repeated processing. Requests with different permissions still require proper isolation.

To Save More Memory, Return to the Test

After tuning caching and scheduling, the weights still occupied about 1TB. Reducing the bits per weight would bring raw 8-bit weights to about 500GB, or 4-bit weights to about 250GB. The freed memory might support more requests and could reduce data movement.

Fewer bits, however, changed numerical precision. I kept the BF16 version as the quality and speed baseline, then checked vLLM’s quantization support list for an 8-bit option supported by the pinned version and B200. Runtime memory also had to include scaling factors and other format metadata, caches, and temporary space.

Each candidate first ran the independent validation questions from Part 23, with particular attention to long documents, questions near the boundary of correctness, and tool formats. Then we repeated the load test at the same concurrency. Speed gains and quality changes would appear in the same record. Any method requiring calibration material would include English, Chinese, code, long documents, and difficult questions.

The first release would remain BF16. An 8-bit version would become a deployment candidate only if it preserved quality and improved sustained throughput. A 4-bit version would get its own comparison in a later round. Any numerical-format change would also need a final release check together with its serving configuration.

Quantization: Representing weights or other tensors with fewer bits to reduce storage and computation-related overhead.

One Replica Is Measured. How Many GPUs Does the Service Need?

In this round of load testing, the 16-GPU replica sustained 8 requests per second within the target waiting time. The input and output length distributions were saved beside that result. A question producing 100 tokens and one producing 2,000 tokens consume very different amounts of resources.

The expected busy-period traffic was 20 requests per second. Instead of planning to run every replica at its measured limit, I set normal load at 70% of measured capacity to leave room for variation. Each replica would handle 5.6 requests per second, giving a working-replica count of:

Round up [20 ÷ (8 × 70%)] = 4

Four working replicas occupied 64 GPUs. With one more replica ready to replace a failed one, the total became five groups and 80 GPUs, or ten eight-GPU servers.

Now trace the complete diagram. A user request passes the gateway checks, and the router assigns it to an available replica. That replica’s two servers execute the model in sequence and return the output. Four groups serve different requests; the fifth, marked as spare, stands ready to take over after a failure. Zoom in on one group, and we are back to the two machines from the beginning: eight GPUs in each, one handling the earlier model layers and the other the later layers.

After main training, these ten servers would come from the original active pool, while the rest continued research and evaluation. The 20 training spare servers would retain their repair and replacement role. If the next round needed all 5,120 GPUs for training at the same time, serving would need a separate allocation, or we would have to adjust the training scale and schedule.

Finally, we needed to replay real request traces across the five groups and check peaks and failover. The 8 requests per second applied only to that load-test input mix. A change in the share of long documents, answer length, or user behavior would trigger a new capacity calculation.

I pasted the manual into the chat window again. What happened on screen was still simple: a brief wait, then words appeared. Behind it, we now had a complete path: who accepted the request, which two machines handled it, and how the next request would reach another group when traffic grew.

The model had finally moved from the training directory into users’ hands. Next, I would put the running cost of these 80 GPUs beside the machines, people, and time invested earlier, and ask what it really cost to answer a request and complete another round of improvement.