[Training a 500B LLM from Scratch] 22/25: How Does a Model Learn from Its Own Attempts?

After the last round of fine-tuning, the model had become better at answering as requested. I gave it another problem: “A car travels 60 kilometers, then another 90 kilometers. The whole trip takes three hours. What is its average speed?”

The first answer was 50 kilometers per hour. I generated another, and it became 75. Two more attempts gave 50 and 45. With all four answers side by side, the problem was concrete: the model could find the right path, but did not follow it reliably.

I saved the four answers. Since we could check which were right, perhaps the model could learn from its own attempts. The next job was to connect “this answer is better” with “a better answer is more likely next time.”

Four Answers. How Do We Score Them?

The total distance was 150 kilometers and the total time three hours, giving an average speed of 50 kilometers per hour. The scoring program first identified the answer and unit, then checked the calculation. The four answers received 1, 0, 1, and 0.

The saved files held more than those four numbers. Each contained the full question, the tokens the model actually produced, and the probabilities assigned to those tokens during generation. One correct answer said, “The total distance is 150 kilometers. Dividing 150 by 3 gives an average speed of 50 kilometers per hour.” Another made an error along the way and ended at 75.

All those records would matter later. The score told us how well the attempt had gone. The answer and probability records helped the training program trace how the model had written it.

Here the path diverged from the previous article. SFT placed a reviewed answer in the textbook for the model to practice. These answers came from the current model itself, with feedback drawn from their results. I added the model version to each record so that every attempt could be traced to the parameters that generated it.

Sampling: Generating an answer according to the probabilities the model assigns. Different random values can produce different candidates.

Reward: A score assigned to an attempt, telling the training process which behaviors are worth making more frequent.

Reinforcement learning: Training that obtains feedback from actions and their outcomes, then uses it to adjust future behavior.

How Does a Score of One Find Its Way Back into the Model?

Looking at the score sheet, it was easy to skip precisely the step that mattered. The answer had a score, but the model had 500 billion parameters. Which of those numbers should a one change?

I first put the four answers on the same comparison scale. Their mean score was 0.5. Subtracting it from each score gave +0.5, −0.5, +0.5, −0.5. A positive value meant better than the group's average performance; a negative value meant worse. Each answer now had its own direction, instead of all sharing one average score.

The four sheets in the diagram stand for four complete answers. Ones and zeros show the check results; +0.5 and −0.5 show the differences after subtracting the group mean. Full GRPO also handles signal scaling, compares probabilities under the old and new models, and limits update size. Together, those steps organize feedback into a training objective.

With a direction established, the training program rereads the question and the four answers, calculating the current model's probability for each answer token. Relatively good answers provide a signal to increase the probability of such answers; poorer ones provide a signal in the other direction. Combining these signals with probabilities produces a loss that can be differentiated with respect to the model parameters.

At the loss, we rejoin the familiar training process. Backpropagation calculates how parameters affect that loss. The optimizer then uses the gradients to adjust attention, feed-forward layers, and other trainable parameters. The new parameters change the probability distribution for the next round of answers.

The scorer only needs to know whether 50 kilometers per hour is correct. The training computation graph and optimizer determine which parameters to change. A reward is not an adjustment written directly into a parameter. Many words and abilities share the same parameters, and each update combines records across a batch. The actual effect has to be checked in new answers.

I chose GRPO to organize this round. The TRL GRPO documentation describes within-group comparisons, the training objective, and reward interfaces. It suits tasks like this one, where we can try repeatedly and check the results. The first goal was to connect the cycle: generate, score, update, and generate again.

Relative reward signal: A training signal obtained by comparing an attempt with other attempts at the same question. Here, subtracting the group mean illustrates the direction of improvement.

GRPO: Group Relative Policy Optimization, which constructs learning signals from a group of answers to the same question and their rewards.

Backpropagation: Starting from the training loss, calculating how parameters affect it, and passing those gradients to the optimizer.

All Right and All Wrong Both Give Us Reason to Pause

Further through the same batch, some questions had four correct answers and others had four wrong ones. Under this binary scoring scheme, neither group contained an answer better than its peers. The comparison offered little useful distinction.

That sent me back to the questions. Easy problems were already solved reliably; very hard ones supplied no successful attempt to reinforce. The first training batch needed some questions the model could solve but had not yet mastered consistently, so that its answers could differ meaningfully. For problems it could not solve at all, reviewed demonstrations could come first, followed by another round of practice.

Four answers per question became the initial configuration. When comparing four with eight later, I would give both the same GPU-hour budget and ask whether the additional attempts justified their generation cost. More candidates turn a question from one row on a score sheet into more text to generate, verify, and store.

Why start with GRPO, and where does PPO fit?

PPO commonly trains a value model to estimate future returns and help organize updates. For the current single-turn math and code tasks with checkable results, I chose GRPO, using multiple answers to the same question for comparison and reducing the engineering burden of an additional value model. Once tasks expand to long sequences of tool use, I will compare algorithms based on feedback sparsity, training stability, and cost.

PPO: Proximal Policy Optimization, a reinforcement learning method that organizes updates by limiting changes between old and new policies, often with value estimation.

A Correct Score Does Not Guarantee a Correct Explanation

Returning to the two answers that earned one point exposed a gap in the scoring rule. If it checked only the final 50, almost anything written before it could pass. An incorrect derivation that happened to land on the right number would receive the same score.

I separated the acceptance checks. The final answer carried the main score, units and format were checked separately, and explanations were sampled for review. To train the reasoning process further, I would first collect solutions with verifiable or human-reviewed steps, then add process feedback. The reasoning text a model outputs is a solution we can inspect. We can judge its clarity and correctness, but the text alone does not establish that it fully reveals the model's internal computation.

The issue became clearer with code. Take the sorting task discussed earlier: if tests inspect only the first three numbers, a program that sorts only those three might score well. Training follows the score, and whatever the tests omit becomes a shortcut.

The scorer itself therefore joined the checklist. Tests added different lengths, duplicate values, and empty inputs, along with held-out tests the model could not see and random inputs to probe boundaries. Programs ran in an isolated environment with task-specific limits on time, resources, and network access. Test files lived where the program could not modify them. Every score needed a trace of what had actually run.

Anthropic's 2026 Reward Seeker research discusses models exploiting environmental loopholes to pursue rewards. The sorting test made that issue tangible: every point from the scorer tells the model which paths are worth repeating.

After changing the scorer, I ran the saved old answers through it again. Results under the new and old rules stayed separate, so that a changed exam would not be recorded as an improved model.

Reward hacking: Exploiting gaps in scoring rules to earn higher scores without a corresponding improvement in the actual task.

Verifiable reward: A score assigned after checking a result through calculation, tests, or explicit rules.

What If the Rain Explanation Has No Single Correct Answer?

Math lets us check a number; sorting code lets us run tests. Returning to the previous article's rain explanation, however, two answers could both be correct. The difference might be how well they suit an elementary school student or follow the request for three sentences.

I gave a reviewer two answers to the same question. A was long and omitted the cooling of water vapor. B was shorter and explained the conditions and process fully. Using criteria written beforehand, the reviewer chose B and recorded why. For the next comparison, we swapped display order and hid version names to see whether the choice remained stable.

The resulting data contained a pair of answers and a judgment about which was more suitable. For open-ended tasks, I started with DPO and these reviewed preference pairs. The public DPO implementation takes the question, preferred answer, and less preferred answer as its basic inputs, connecting preferences to the relative probabilities of the two answers.

The first experiments therefore split into two branches, both starting from the same SFT version. Math and code used GRPO with verifiable rewards; open-ended answers used DPO with reviewed pairs. I would evaluate them separately before deciding the order in which to combine them. A change in an ability could then still be traced to the training round that produced it.

If we later needed the model to keep generating open-ended answers, score them automatically, and update repeatedly, I would consider training a reward model. That would add another scorer to calibrate and maintain. For now, the human criteria and paired data came first.

Preference data: Data recording which answer to the same input better meets the requirements.

DPO: Direct Preference Optimization, a method that trains a model using the relative preference between better and worse answers.

Once Batches Run, Where Does the Time Go?

With the average-speed problem working through the cycle, I planned the first batch of 1,000 questions, with four answers each. At an average of 1,000 output tokens per answer, generation would produce about four million output tokens. Question inputs, unfinished answers, and retries would consume more time.

Now the two kinds of work became visible. The first stage lets the model write answers token by token. The second reads those answers and scores back in to calculate an update. Watching GPU activity only during updates would miss generation, verification, and model-switching overhead.

For the first round, the two stages would take turns. Fix the current model version, generate and score a batch of answers, then pass them to the training program for parameter updates. The new version would generate the next batch. Version numbers connected the whole cycle, identifying which model produced an answer and which update used it.

Generation would first be validated on one 16-GPU replica: two eight-GPU servers hosting the 500B model together. Part 24 will explain how it processes input and produces output step by step. Updates would return to the validated training system, loading the records and calculating loss and gradients. The stages would initially share the same resource pool at different times, with model loading and switching included in elapsed time.

I split the cost ledger into generation, verification, switching, and updates. If generation was slowest, I would compare the benefit of more generation replicas. If switching consumed too much time, I would revisit batch size and possible overlap between the stages. Answer length was recorded too: raising average output from 1,000 to 2,000 tokens would take total output straight from four million to eight million. The extra text needed to earn its cost through better performance.

How do generation and updates share one resource pool?

The first 500B rehearsal saves the current policy version, exports inference weights, and exits training processes to release GPU memory. The inference environment generates and saves answer records, then exits. The training environment loads the same policy and optimizer states to compute updates. Slurm schedules the two stages separately, with the same version identifier on model files and answer records.

The 16-GPU generation candidate uses TP8 within each machine and PP2 across two machines. Before enabling it, I validate architecture support, weight conversion, and peak memory, then generate four candidates for each of 1,000 questions and record the time. Updates retain the validated training parallel configuration, with answer lengths and batch size checked again. Weight reads, format conversion, and cold starts all count toward the cycle.

As of November 2026, TRL has published an experimental asynchronous GRPO implementation. Its example distributed stack uses FSDP2, which differs from this series' Megatron parallel configuration. I will first validate overlap between generation and updates on a small model, then put the 500B implementation through parallelism, weight synchronization, and recovery checks. An asynchronous approach records the policy version used for generation and controls update bias from answers produced by older versions.

Policy: Here, the probability distribution the current model uses to produce answers.

Online data: Data generated by the current or a recent model version and subsequently used for training.

Change the Test and See What It Learned

After an update, the average-speed problem was still the first thing I checked. But it had participated in training. Getting it right could only suggest that the cycle was working. I changed the numbers and wording, then rearranged the conditions to see whether new problems improved too.

The rain explanation, ordinary questions, and missing-information tasks stayed alongside it. After becoming more reliable at math, would the model write long derivations for every simple question? When material was missing, would it invent conditions just to produce an answer? Those changes went into the record with accuracy.

For each candidate, I kept task success rate, answer length, tool errors, examples of serious violations, and total GPU-hours. A scorer upgrade meant rescoring old answers. A model update meant answering independent questions again. Recording the two separately made the effect of training visible.

I kept the first four answers on the opening page of the experiment log. The sequence 50, 75, 50, 45 had acquired four rewards, individual relative signals, a training loss, and a parameter update. The path from “a good answer” back into the model now had a concrete step at every turn.

Soon there would be several models on the desk: one stronger at math, one clearer at explanations, and another faster to answer. Choosing a version to give users would require a different set of judgments. In the next article, I will put those candidates on the same acceptance checklist.