[Training a 200B LLM from Scratch] 10/25: How Can Existing Models Help Us Write Training Material?

In the previous articles, we cleaned and organized the training material so the model could read it. The next question is what that material should teach. We want the model to produce a result and connect it to the evidence and steps behind it. Some source documents give only conclusions, and some programs leave their workings unexplained. We can supplement them with material that walks through the process. Existing models can help write it; controlled training comparisons will tell us whether it helps the new model learn.

Suppose we find a short passage: “I read 2 pages on the first day, 3 on the second, and 4 on the third. I read 9 pages in total.” It gives the result. We can also show how to reach it, step by step.

We could expand it: “After the first day, the total is 2 pages. On the second day, add 3 to get 5. On the third day, add another 4 to get 9.” Both passages describe the same event, but the second also preserves the calculation. When preparing material for a new model, we can keep the original and add explanations, then see whether the combination helps it learn better.

An existing model can help with this work. We call the already trained model that writes the material the teacher, and the 200B model we are training from scratch in this series the student. The teacher adds explanations to the source material. We check them and put the accepted text into the student's reading material. We then test the student on unseen problems, comparing runs with the same training budget to judge whether the additions were useful.

Material produced by the teacher is called synthetic data. Training the student on content supplied by the teacher is one form of distillation. This can apply to natural language, mathematics, and code. This article follows one piece of material through that process, starting with the steps behind a result.

Synthetic data: Training material generated by models or other tools. Here, we use an existing model to help write explanations, then check them.

Distillation: Training a student with learning signals such as a teacher's outputs. Here, we use verified teacher text.

Why does this material include code?

Return to the reading example. The daily page counts form one sequence: 2, 3, 4. The running totals form another: 2, 5, 9. If we kept this record for a year, adding up all the earlier days again each time would be tedious. We can write a small program that remembers the total, adds each new number, and saves the updated total.

The program performs the same operation that we just described in ordinary language. Code is also text, with stricter rules for expressing operations. We want a general-purpose model to explain ordinary passages, understand other people's programs, and help write programs. Our training material therefore includes code and documentation alongside Chinese and English text. The tokenizer converts all of it into tokens the model can read.

Source code is often concise. It leaves out why it was written that way and what happens at each step. That gives the teacher a specific job: turn suitable code into instructional material that shows the program, its execution, and its result together.

Material like this already appears in real pretraining recipes. The Domyn Edge dataset documentation, released on October 20, 2026, describes adding reasoning traces to most Python, Java, and C++ documents for pretraining. It turns the idea of adding a process alongside code into a data production task. We can borrow the approach and test it on our own tasks; that enhanced code is awaiting release once source attribution is complete.

Part Seven introduced synthetic data and distillation, reserving 10B tokens for these explanations in our 4T training recipe. That is 0.25% of the total. Combined with 590B tokens of original code and documentation, it makes up the 600B code allocation. Part Eight covered cleaning the material, and Part Nine covered turning it into training inputs. Now we can make this reserved portion concrete.

Token: A small unit of text produced by a tokenizer. Ordinary sentences and programs are both split into tokens and converted into IDs that the model reads.

How does the teacher turn a result into a lesson?

We first give the teacher a small program with a defined purpose. It takes a sequence of numbers and returns the running total through each position. For 2, 3, 4, the result is 2, 5, 9. With no input numbers, the result is empty. Page counts help us understand the operation; the program can also handle other numbers that need running totals.

The following function was written for this example. You can keep reading without knowing Python: it does just three things: remember the total, add the next number, and save the new total.

def running_totals(numbers):
    total = 0
    result = []
    for number in numbers:
        total += number
        result.append(total)
    return result

The data team gives the teacher the original program and explains what to add:

Turn this program into a short lesson. First explain the problem it solves. Then use the input 2, 3, 4 to show how the total and result change at each step. Finally, explain what happens when there are no input numbers. Keep the original program, and make every step of the explanation match its execution.

The teacher now has a specific task. It must explain the steps left implicit in the short program and provide an example we can check. Here is an illustrative lesson written to those instructions:

Running totals

Given a sequence of numbers, return the running total through each position, including the current number. For example, reading 2, 3, and 4 pages on successive days gives running totals of 2, 5, and 9 pages.

Initially, the total is 0 and the result list is empty.

Read 2 and add it to the total to get 2. Save it, giving the result [2].

Read 3 and add it to the previous total of 2 to get 5. Save it, giving the result [2, 5].

Read 4 and add it to the previous total of 5 to get 9. Save it, giving the result [2, 5, 9].

Each step saves the total through the current position. With no input numbers, the loop runs zero times and returns an empty list.

The complete lesson inserts the original program from the expandable section after the paragraph containing the page-count example, then continues with “Initially, the total is 0.” The program, input, explanation, and conclusion now belong to one document. This small example shows what the teacher adds. Actual production uses the teachers and access methods selected in Part Seven for the intended training use. The team selects permitted code and documentation, then requests the appropriate explanations.

Once the lesson is written, check it. The data team runs the program to confirm that it returns 2, 5, 9, then checks each running total in the explanation. It also runs the empty-input case. As the collection grows, tests can include negative numbers, zero, and repeated values. The program should still satisfy the running-total rule: the difference between two adjacent totals must equal the new number added at the current position.

The explanation needs its own check. A program can run correctly while its description confuses the current input with the running total. Code can check the numbers; sampling and review check whether the prose is accurate, clear, and missing any conditions. The lesson enters the accepted collection only when both parts agree.

This pattern of generating and then checking also applies to ordinary language material. In research released on October 19, 2026, NVIDIA and AI4Bharat translated curated English documents into Indian languages, assessed translation quality, repaired low-quality content, and retained source-document records. Their released synthetic corpus contains about 2.83T tokens across 22 Indian languages; that is the size of the corpus. The study then selects material, mixes it with natural text for continued pretraining, and tests its value through controlled experiments. Code requires checks on execution; translation requires checks that the original meaning survives. Each needs its own verification method.

Then production can expand. We need different programs, difficulty levels, and ways of explaining them. Merely replacing 2, 3, 4 with other numbers quickly produces many similar lessons. The deduplication and source tracking from Part Eight still apply, and material derived from the same source stays grouped together for management.

What does the student actually read?

The student still reads a document. Its title is “Running totals,” and its body contains the task, program, explanation, and conclusion. The data team attaches a tracking label recording the source, teacher version, and checks performed.

The body and label are stored separately. In the record below, text holds the complete lesson and metadata holds the tracking information. During training, the reader sends only the body to the tokenizer. The model sees both the program and its natural-language explanation. Source IDs and review records stay in the data system.

{
  "text": "Running totals\n\nGiven a sequence of numbers, return the running total through each position, including the current number. For example, reading 2, 3, and 4 pages on successive days gives running totals of 2, 5, and 9 pages.\n\nProgram:\ndef running_totals(numbers):\n    total = 0\n    result = []\n    for number in numbers:\n        total += number\n        result.append(total)\n    return result\n\nInitially, the total is 0 and the result list is empty.\n\nRead 2 and add it to the total to get 2. Save it, giving the result [2].\n\nRead 3 and add it to the previous total of 2 to get 5. Save it, giving the result [2, 5].\n\nRead 4 and add it to the previous total of 5 to get 9. Save it, giving the result [2, 5, 9].\n\nEach step saves the total through the current position. With no input numbers, the loop runs zero times and returns an empty list.",
  "metadata": {
    "record_id": "example-001",
    "source_id": "self-authored-running-totals-example",
    "teacher_version": "illustrative-output",
    "review_record": "illustrative-review-record",
    "family_id": "running-totals-example",
    "split": "train"
  }
}

Metadata: Information attached to a document, such as its source, teacher version, and review record. It helps the data team manage the material.

JSON: A text format for structured data. The record above stores the document body and its tracking information separately.

Once the body passes review, count its tokens with the student's tokenizer and add it to the collection. How much text the teacher generated determines production cost. How many accepted tokens the student actually reads determines training volume. Rejected material still cost money to produce, so both records matter.

The 10B allocation now has a practical meaning. If accepted lessons average 2,000 student tokens each, 10B corresponds to five million lessons: 10,000,000,000 divided by 2,000. That calculation estimates production scale. The amount we can retain depends on each batch's length, acceptance rate, and deduplication results. Starting with a small batch tells us how much it costs to obtain one million accepted tokens.

The mixture from Part Nine schedules when the student reads them. At the reserved share, every 10B training tokens includes 25 million tokens of teacher explanations. Across the full 4T plan, the explanations account for 10B, already included in the total. Original code and documentation still account for 590B, making 600B together. Compute used by the teacher to produce the material is counted separately.

How does the model learn from this lesson?

Return to the running-total lesson. A person reads that the total changes from 2 to 5. The model splits the same text into tokens and predicts what comes next from what came before. It reads the input 2, 3, 4 and the initial total of 0. The program and explanation unfold under those conditions.

During training, the lesson supplies the actual text that follows. The lower the probability the model assigns to that text, the larger its prediction loss. The training software uses the loss to calculate how to adjust the parameters and updates the model's weights. When the next document arrives, learning continues with those updated weights.

The task, code, steps, and conclusion all belong to this pretraining document and all provide learning signals. Splitting long documents, packing short ones, and handling document boundaries follow Part Nine's rules. At this stage, we are training a general-purpose base model using complete documents. Teaching it to answer conversational requests will have a separate post-training plan.

The teacher's job here is to supply the material. During training, the student reads the saved lessons. Running totals show how one document provides a signal; many different lessons expose the student to more patterns and relationships. The next step checks whether it can use what it learned on new material.

Prediction loss: A measure of how much probability the model assigns to the actual next tokens. The training software uses it to adjust the parameters.

Weights: Parameters that the model reuses across different inputs. Training changes and saves them.

How do we know these explanations are worth producing?

We expect fuller explanations to help the student make connections. We can test that expectation directly. Starting from the same training checkpoint, two runs read the same number of additional tokens with the other training settings held constant. One reads teacher explanations according to the recipe. The other replaces that portion with an equal amount of original code and documentation. Both keep the same total code share, along with the same shares of Chinese, English, and mathematics.

Then ask the student to work with unseen material. Can it explain new programs, follow new execution paths, or complete new coding tasks? Have its Chinese, English, and mathematics abilities changed? We are training a general-purpose model, so a local change also needs to be judged by overall performance.

These test problems are reserved before material production begins and separated from training material by source and task family. A development validation set helps find problems and refine the material. The final test set stays untouched until the plan is settled. Small models help rule out clearly ineffective production methods. Checkpoints of the target model then test the actual gains, while the team tracks the cost of teacher generation, verification, and student training.

The ICLR 2026 paper Front-Loading Reasoning, first posted in 2025, compares adding reasoning data at different stages using 8B models. It provides a basis for studying this kind of material during pretraining. Our 10B allocation is the portion reserved for this project's experiment. Our own results will determine whether to retain, change, or expand it.

This article began with a passage that gave an answer but left the process unexplained. A teacher helps write out that process, the data team checks it, and the student learns from it. The question we need to answer is whether giving some training time to these explanations helps the same 200B model learn better. Connecting production, verification, training, and unseen tests gives us a way to answer it step by step.