[Training a 200B LLM from Scratch] 9/25: How Do We Turn the Material into a Training Recipe?

In the previous article, we cleaned the material. Now the data team hands over a complete sorting tutorial: the first chapter explains the problem, the second develops the algorithm, the third provides the code, and the fourth checks the result with tests. It contains paragraphs, formulas, code, and solutions. Our next job is to turn a long article like this into material that the training program can read one segment at a time.

Let us use a set of teaching numbers to see the process. After tokenization, the article contains 12,000 tokens. Its four chapters contain 3,000, 3,500, 3,000, and 2,500 tokens respectively. These counts illustrate the cutting process; an actual article must be counted with a tokenizer. For now, think of a token as a small piece of text.

If we allow at most 4,000 body tokens in each training window at this stage, we must divide the article. Cutting every 4,000 tokens in order gives us:

1. Segment one: tokens 1 through 4,000. It contains the complete problem statement and the beginning of the algorithm's derivation.
2. Segment two: tokens 4,001 through 8,000. It contains the rest of the derivation and the first half of the code.
3. Segment three: tokens 8,001 through 12,000. It contains the remaining code and all the tests.

What began as a tutorial that a person could read from beginning to end is now three training samples. Each keeps its original order. The program converts the text into IDs and lets the model practice predicting the next token. We need to decide where to cut and how much preceding text each sample should carry.

The example immediately reveals a problem: the second segment starts halfway through the algorithm, and the third starts halfway through the code. When the model receives the third segment, the first two do not automatically appear in front of it. The cuts need to work with the content.

Where Should We Cut a Long Article?

First, why cut at all? During training, we choose how long a sequence the model reads at once, then allocate memory and computation around that length. Longer sequences generally require more intermediate results to be retained and processed. With full attention, the number of pairs of positions also grows with the square of the length. We therefore choose a window length suited to the current stage, and schedule especially long material for long-context training.

The example reserves 4,000 tokens for the body. A real training sequence must also include the necessary markers. This teaching configuration explains the cuts; the production sequence length must be chosen through model and memory tests.

We count in tokens. A thousand Chinese characters, a thousand English words, and a thousand lines of code produce different lengths. Tokenization tells us which sentence can fit at the end of a window.

The most direct approach is the fixed-length cutting shown in the figure. It preserves the article's order and makes a continuous data supply straightforward. It can also cut through a sentence, a derivation, or a code block. Large-scale pretraining can use this approach. NVIDIA's GPT dataset implementation samples by document ID and token offset to assemble inputs of a specified length. To recognize paragraphs or chapters, the program needs additional data-processing rules from us.

For this well-structured tutorial, I would first compare another approach: finish each segment at a complete section where possible. The four chapters in the example give us 3,000, 3,500, 3,000, and 2,500 body tokens. All four fit within 4,000 body positions. This preserves the boundaries of the problem statement, derivation, code, and tests. The immediate benefit is that the cuts leave sentences intact.

Next, check whether each sample makes sense on its own. If the test chapter says “call the function from the previous chapter,” bring the corresponding function and necessary input conventions with it, then count again. If a problem and its solution fit together, put them in the same window. When chapters depend heavily on each other and the whole group exceeds the window, schedule a longer training sequence so the model can actually see their relationship in one computation.

The four bodies still total 12,000 tokens. Any added problem statement, function, or explanation must be counted separately, and each sample checked again for fit. We can use remaining space when we get to combining short material later.

If one section alone contains 6,000 tokens, look for subsection, paragraph, or code-block boundaries within it. Put any unit that must remain continuous but is still too long into the long-context material pool. Ordinary continuous prose can be cut further by token count, with a small overlap of preceding text carried into the next window.

For example, keep the 4,000-body-token limit: the first window reads tokens 1 through 4,000, and the second reads tokens 3,501 through 6,000. The second window begins with the last 500 tokens of the previous segment, then continues with 2,000 new tokens. It now has context near the cut, while its view remains limited to its own window.

4,000 + 2,500 = 6,500 cumulative body tokens

The unique material contains 6,000 tokens. Overlap adds 500 tokens to the processing workload, about 8.3%. Markers and padding are counted separately. We can also use those 500 tokens only as context and turn off their prediction loss. They still consume computation and must remain in the ledger. The 500-token overlap is a choice for this teaching example; whether we need overlap, and how much, depends on the material's dependencies and validation results.

The distinction between windows matters. Model parameters accumulate experience through training, but the specific words in one window do not automatically remain in the next window's attention. If we need preceding text, we include it in the current input or use a longer window. Here we are preparing training samples. The KV cache used during inference to retain already-computed context is a separate mechanism for generation.

Finally, keep the original document ID, starting and ending token positions, and overlap range for each slice. We can then trace a slice back to its source and see which content was processed more than once. After dividing long material, we have another job for short material: fit several short pieces into one training sequence. Cutting long articles handles material that will not fit; combining short articles uses the space that remains.

Training window: The content the model can read in one sequence computation. The 4,000 in this article is the example's body-token limit.

Context: The preceding text available to the model at the current prediction position.

Overlap: A passage from the original document included in adjacent training windows to give the later window preceding context.

Why Can the Same Text Have Different Token Counts?

A long article's cuts ultimately fall on tokens. Now zoom in on three fragments from the tutorial to see how this ruler counts. We process them with the public GPT-2 tokenizer and the public Qwen3 tokenizer. These two public tools make the differences visible. The results below came from a local run on October 21, 2026, with no chat template or special markers added.

Both tokenizers split the first sentence into Keep, duplicate, values, and .. The spaces at the start of the middle two pieces belong to those tokens. For this sentence, the counts match. The code shows where the rulers differ: GPT-2 splits sorted(xs) into s, orted, (, xs, and ), while Qwen3 produces sorted, (xs, and ). These pieces can cross the word or symbol boundaries we are used to. Their job is to represent the original text using pieces already in the vocabulary.

The last row is useful too: Qwen3 produces more tokens in that case. When choosing a tokenizer, we need to compare Chinese, English, code, numbers, and formulas together. One example with a shorter encoding cannot tell us how much computation the entire training set will save.

Hugging Face's description of the tokenization pipeline separates text processing, splitting, mapping to vocabulary IDs, and adding special markers. In our sorting tutorial, spaces, indentation, and symbols matter. We must be able to recover the original text after processing. All six encodings in the table passed a complete decoding round-trip check. Some tokens contain only part of a character's bytes, so an individual token may look incomplete when displayed alone. The full sequence can still reconstruct the text.

Token: A piece produced when a tokenizer processes text. It may be a whole word, part of a word, a symbol, or a sequence of bytes.

Vocabulary: The list of pieces and their IDs used by the tokenizer.

What Happens When the IDs Enter the Model?

Continue with sorted(xs). In this Qwen3 example, the three pieces have IDs 28298, 53322, and 8. An ID is used for lookup. A larger ID simply indicates a different position in the vocabulary.

The model receives an ID and retrieves a row of numbers from its input table. That row is the vector we discussed in Part Four. Three tokens therefore become three rows of vectors, which then enter the first attention layer. One token ID is an integer; one token's vector contains many numbers. The figure draws them as small labels and long bars to make these two steps clear.

This also explains a practical choice. When we reuse existing model weights, we begin with the model's tokenizer and special markers. The model has already associated each ID with a row in its input table. If a new tokenizer assigns different text to the same ID, that association changes. Extending or replacing the vocabulary requires corresponding changes to the input and output layers and further training, so we should treat it as a separate experiment.

Training from scratch lets us compare new vocabularies. I would first fix a sample covering Chinese, English, code, and mathematics, then compare the candidates' total token counts, encoding lengths by category, reconstruction results, and vocabulary sizes. A larger vocabulary also requires more space and computation in the input table and output prediction stage. We need to weigh those costs against the benefit of shorter sequences. Once we choose, we freeze the vocabulary files and processing rules, then recount the entire inventory with them.

When integrating an existing model, we can load AutoTokenizer and weights from the same model version and record them as a pair. Our 4T ledger uses this same ruler from the final student model.

Token ID: The integer assigned to a token in the vocabulary, used for lookup.

Embedding: The step that converts a token ID into a vector, and the trainable parameter table used to retrieve those vectors.

Vector: A sequence of numbers. Here, each token retrieves a sequence of numbers representing its features as it enters the model.

How Do Short Documents Fit into One Training Sequence?

After cutting long articles, some pieces leave room in their windows. Many other documents are short to begin with. We can organize these short materials into the same training sequence. Start with a small row of eight positions to see how it works; actual training uses longer sequences.

Suppose document A contains three body tokens and document B contains two. Add an end marker after each. Placing them in separate eight-position sequences requires sixteen positions, of which only seven hold body text or end markers. Put both into one eight-position sequence and only one empty position remains.

This process is usually called packing. In an implementation with the same fixed sequence length, it reduces the padding positions allocated to these documents. The actual throughput gain must be measured on the chosen training backend. A sequence can contain several documents, each with its own boundary. Standard Megatron pretraining already concatenates documents during sampling. The Megatron Bridge documentation specifically distinguishes this pretraining behavior from the packing switch used during fine-tuning. We need the method supported by our training path.

Once the documents are together, can attention cross their boundaries? The recipe needs an explicit answer.

A sorting tutorial and a weather report happen to occupy neighboring positions. For independent material like this, our initial plan uses attention within each document: the weather report reads its own preceding text, and the sorting tutorial reads its own. The two small triangles in the figure show each reading from left to right. The blank area between them shows that they do not exchange information. An end marker tells the model where a document ends; the attention mask in the program must enforce the reading boundary.

Learning targets follow the boundaries too. We train the model to predict the next token within a document, including its end marker. We do not use the preceding document to predict the first token of the next independent document after that marker. This cross-document prediction and padding positions are excluded from the loss. Position IDs restart within each document. NVIDIA's GPT dataset implementation handles positions, attention, and loss separately, and engineering acceptance should check each separately too.

Related material can use another arrangement. Functions, tests, and documentation from the same code repository need to refer to each other. We first organize them into one ordered training unit with file markers, then allow information to pass within that unit. Isolating independent documents and grouping related files serve different learning tasks.

Training sequence: An ordered sequence of tokens fed into the model at one time.

Packing: Organizing material into training sequences to fill the available positions more fully.

End marker: A special token that marks the end of a document.

Attention mask: The rule that specifies which positions a given position can read.

Loss: A number measuring the difference between the model's prediction and the training target. Training uses it to adjust parameters.

Turn the 4T Ledger into Daily Quotas

The documents are now sequences that we can train on. Next, we decide which material pool supplies their content.

We keep the capacity plan from Part Seven's from-scratch example, with a workload ceiling of 4,000B tokens. Unique inventory and cumulative reading volume are already separate. Dividing each cumulative amount by 4,000B gives the initial quotas:

For example, a forthcoming block of 10 billion, or 10B, tokens allocates 6.25B, 2B, 1.475B, 0.25B, and 0.025B to those pools in order. The five amounts add up to exactly 10B. Native code and teacher code together account for 15%, consistent with Part Seven.

Allocation is by token. If a long English article contains ten thousand tokens and a short Chinese article contains one hundred, taking one hundred documents from each gives a hundredfold difference in learning volume. One implementation is to organize each pool into equal-length training samples counted on the same basis, then mix them by quota. With variable lengths or padding, keep counting actual tokens and correct the sampling. Megatron's blended dataset organizes reads through sample indices and weights. The project still needs to verify how those sample weights translate into the token ledger.

Shuffle within each category before reading. The Chinese pool receives 800B from an inventory of 400B, an average of two passes. How often a particular record appears depends on the sampling method and stopping point. Two passes on average do not require every document to appear exactly twice. Training logs record cumulative tokens by category, unique documents covered, and repeat exposure.

The 4T remains the workload ceiling for the from-scratch example. A route that reuses existing weights sets its own incremental quotas from the current model's capability gaps. Both routes use the same counting and recording methods.

Sampling weight: The relative share that determines how much content the training program draws from each category.

Random seed: A number controlling the starting point of a random process. Together with the same data, program, and sampling state, it can reproduce the same order.

What Evidence Should Change the Quotas?

The percentages in the table give the first experiments a starting point. If the model is weak at code, we can compare two recipes: A keeps code at 15%; B raises code to 20% and reduces general English by five percentage points, with everything else unchanged. Both start from the same checkpoint, process the same number of effective tokens, use the same training settings, and record their GPU hours.

This experiment changes one thing: some opportunities to learn general English become opportunities to learn code. We then check code, Chinese, general English, and mathematics together. How much code improves, and whether general reading gets worse, determines whether the next training block should use B. The comparison therefore includes the cost of increasing one category.

Use smaller models to eliminate clearly unsuitable recipes, then validate at stage checkpoints of the target model. Record variation in the results of each round and retain an independent final test set. Part Seven's comparison between new and repeated material belongs in this same experimental framework.

NVIDIA's mid-training study, published on October 15, 2026, offers a useful reminder: proxy metrics on a base model may not reliably reflect performance after post-training. The team therefore used a fixed short SFT run to compare recipes. For the coding assistant we want to build, we can likewise run the same short post-training stage on each candidate and compare actual tasks. Falling training loss is one signal; improvement in the intended use determines whether a recipe deserves more training.

Quotas can change by stage. Each adjustment the team approves creates a new version, specifying the checkpoint and cumulative token count at which it takes effect. The quota already spent in each category stays in the ledger. The remaining plan is rearranged within the remaining budget.

Checkpoint: Files that save the model and training state at a particular time, used to compare, resume, or continue training.

SFT: Further training with example inputs and target answers so the model learns how to complete tasks as requested.

What Do We Hand to the Training Team?

At this point, “the data is ready” has a concrete meaning. Our delivery package contains five things:

Align the counting rules before the main run. Part Seven's 4T supports material and compute planning. During execution, record body tokens, special markers, padding positions, and positions contributing to loss separately. Feed the actual processing volume and throughput back into Part Two's calculation to obtain the revised schedule. Differences caused by end markers and boundary handling become numbers we can check here.

Finally, run a tiny acceptance check: fix two documents and verify text reconstruction, correct IDs, whether documents can read each other, and which positions contribute to loss. Then read a small amount, save the state, restart, and check that the next samples match uninterrupted execution. With these two checks, the sorting tutorial becomes material the training program can use reliably.

We now have an executable recipe: fix the mapping from text to IDs, decide how documents form sequences, mix material by token quota, adjust shares through controlled comparisons, and save the complete version and reading state. Material quality, model capability, and the computation actually spent can then be checked against the same record.