[Training a 500B LLM from Scratch] 23/25: How Do We Know the Model Is Ready?

Three versions remained in the training directory. A was the baseline saved after supervised fine-tuning. B and C had also gone through the latest round of reward training. All three could answer questions in the chat window. B was more direct with math problems, C gave fuller explanations, and A was the familiar version we kept for comparison.

Choosing one to release made the question harder. A few pleasing answers could not tell me which model to put in front of users. I needed to know what the new version could now do correctly, and what it had lost along the way.

For the moment, all three stayed where they were. I opened a blank spreadsheet and began filling the leftmost column with questions our first users would ask.

Put the Users’ Questions on the Test

This 500B model would first serve as a general text assistant. Users might paste in a manual and ask for the steps to follow, bring an awkward paragraph for rewriting, ask about math or code, or work out what they need over several turns.

Those activities gave the test five categories: answering from supplied material, writing and rewriting, math and coding assistance, clarification across multiple turns, and handling missing information and boundaries. Each needed English and Chinese tasks, short and long inputs, and a range of difficulty. The last category mattered especially: when the material did not contain an answer, the model should explain what was missing.

We started with just 200 questions, 40 in each category, to get the scoring process working. One reading task supplied an equipment maintenance guide and asked, “When this fault occurs, what should we check first?” The model wrote a fluent answer, but placed the second step before the first.

“Reads well” clearly was not enough. I broke the question into four checks: whether the cited information was correct, whether it answered the question, whether the steps were in order, and whether it added any action the manual did not require. With these four checks, disagreements between reviewers became easier to locate.

Math answers could be checked; code could be tested. The code execution environment was separate from the scoring program, with limits on time and resources, and the model could not modify the tests. For open responses, a model judge performed an initial pass, followed by human review of difficult examples. We hid version names and randomized answer order to check whether length and tone were swaying the scores.

These 200 questions helped us build the measuring instrument. Since the team had repeatedly examined both the questions and the scoring rules, they became development material. Choosing a model would require a separate validation set.

Blind review: Evaluation in which reviewers do not know which model produced an answer, reducing the influence of version names and other identity cues.

Keep One Test Sealed Until the End

For validation, we prepared 1,000 questions in each of the five categories to compare A, B, and C. The evaluation lead held a separate, sealed acceptance set to run only after the candidate was frozen.

The three purposes had to stay distinct. Development questions could be inspected repeatedly to improve the model and the scorer. Validation questions helped us choose a version. The final acceptance set checked whether the selected version met the release requirements. Once a question was shown to the training team for analysis and improvement, we recorded it as development material and added fresh questions for the next sealed acceptance test.

We separated sources as well. Code tasks were split by project or problem family, and long texts by document source. If a manual had appeared in training, asking a differently worded question might still test the same passage. The deduplication records from Part 8 became useful again: we traced provenance instead of comparing only how similar the sentences looked.

The first three sets in the diagram support improvement, selection, and final acceptance respectively. Following the lower arrows, we freeze the candidate, open the sealed test, review the results, and only then begin a limited rollout.

Public benchmarks stayed in the report so we could compare our model with others. I pinned the version of the EleutherAI evaluation harness, saving the data version, prompt templates, sampling settings, and scoring code alongside it. Our new questions and the public scores appeared side by side. Each later model change could be checked against the same record.

Acceptance set: A collection of tasks that checks, under fixed rules, whether a model meets its targets before release.

Evaluation contamination: Test material or closely related content entering training, causing test scores to overstate the model’s ability on new tasks.

B Answered Twenty More Questions Correctly, Yet Stayed Behind

When the results arrived, I put the math and Chinese long-document columns next to each other. A scored 70% on math, B 72%, and C 75%. On long documents, A scored 80%, B fell to 75%, and C rose to 81%.

B’s two-point gain deserved a closer look. The question-level record showed that it corrected 80 questions A had missed, while getting 60 of A’s correct answers wrong. Start with 700, add 80, subtract 60, and the result is 720.

The 80 gains and 60 losses pointed to different changes in ability. A single 72% score compressed them into “20 more correct answers.” The long-document test was more direct: B answered 50 fewer questions correctly. Our first users needed the model to read manuals and find conditions. Those five percentage points would affect their everyday work.

Before seeing the scores, I had set the release rule: the target capability needed clear evidence of improvement, while the risk of regression in other critical categories had to stay within 2 percentage points. B’s long-document score had crossed that line, so it went back for correction. A remained available, and C moved to the next check.

C looked better, but a fresh set of questions might move the scores in either direction. I asked for that uncertainty to appear in the evaluation record too: compare the old and new results on each question, then estimate the range in which the improvement might lie. A range entirely above zero would give us stronger grounds to say the target capability had improved.

On math, C corrected 100 wrong answers and lost 50 previously correct ones, for a net gain of 50. A paired approximation put the improvement at roughly 2.6 to 7.4 percentage points. On long documents, the net gain was 10, with a range of about −0.6 to 2.6 percentage points. The math improvement had support, and the risk of long-document regression stayed within our agreed tolerance. The remaining three categories still needed individual checks. These two columns could not stand in for them.

Paired comparison: Analyzing two models’ results on the same questions together, reducing interference from differences in question difficulty.

Confidence interval: A range estimated from a sample and a statistical method to express uncertainty about a change in a metric.

How do question-level changes become a statistical range?

For 1,000 independent questions, let a be the number corrected and b the number newly answered incorrectly. The difference is d = (a − b) ÷ 1,000. The standard error for a paired normal approximation is √[((a + b) ÷ 1,000 − d²) ÷ 1,000]. Using d ± 1.96 × standard error gives an approximate 95% interval.

B corrected 80 questions and lost 60, a difference of 2 percentage points. Its interval was about −0.3 to 4.3 percentage points, crossing zero. C’s math results, with 100 corrections and 50 losses, gave about 2.6 to 7.4 percentage points. Its long-document results, with 40 corrections and 30 losses, gave about −0.6 to 2.6 percentage points.

A formal report can estimate intervals by resampling questions or task families. If a group of questions simply changes the numbers in one template, resample at the template-family level to preserve the dependence between questions. Fix the next batch size and analysis points before adding questions, rather than repeatedly testing for release after every few additions.

For this release, the lower bound of the paired 95% interval had to be above −2 percentage points in every critical category. For the target capability, it had to be above zero. Serious safety issues were addressed individually.

A Polished Answer Can Still Take the Task Somewhere Else

The capability table had brought us to C, but evaluation was not over. In a reading task, external material might contain an instruction to ignore the user’s question and do something else. If the model followed it, a good answer to the earlier maintenance question would offer little protection.

We therefore built safety cases around real entry points: insert distracting instructions into supplied material and check whether they redirect the task; include fields with different access permissions and inspect what the model reveals; make tool calls encounter an unclear target or an execution failure and check whether the model exceeds its authority or stops to explain the problem.

Anthropic’s 2026 investigation into unintended model actions helped inform these checks. For our own model, each scenario still needed an explicit definition of allowed behavior, complete inputs and outputs, and tool records. A serious issue meant fixing the model, prompts, or system boundaries, then adding the relevant scenarios to the regression set for every later version.

We also narrowed the initial release to text answers. External writes and execution stayed in a controlled environment until each capability passed its checks. The final safety record’s “0 unresolved issues” would then correspond to a defined test scope and a specific set of product permissions.

Regression set: A fixed collection of tasks rerun after model or system changes to detect whether previously resolved problems have returned.

Once the Scores Pass, How Long Does the User Wait?

With C still on the shortlist, I connected it to the trial environment and pasted in the same manual. The answer was correct, but users would care about two more things: how soon text appeared, and whether it kept stalling as they read.

We had written the service targets in advance. For ordinary short questions, 95% of requests should begin producing output within 2 seconds, with a typical interval near 50 milliseconds per token during generation. Long-document requests would record input length and waiting time separately. In everyday terms, 95% meant that roughly ninety-five out of a hundred requests had received their first output by that point.

Load tests began at low traffic and increased gradually. Each run used the same input and output length distributions, recording latency, errors, and GPU cost. Speed with just one question in flight could look impressive. The machine count would depend on whether we could keep the wait acceptable while questions kept arriving.

Trying the same question several times changed the bill too. If a user received one answer, we reported success from one generation. If a feature used a verifier and retries, we measured retry count, total waiting time, and cost together. I added this rule to the record so that “correct after eight generations” would not slip into a single-answer score.

The records for all three versions gradually came together in one table. A few rows appear below; the full report retained all five task categories and every safety check:

C’s 1.8 seconds stayed under the 2-second target. Generation intervals, other tasks, and error rates remained in the report for review. Any missing item had to be completed. Only after every condition was met would we freeze C as the final acceptance candidate.

Tail latency: The waiting time experienced by the slower portion of requests, such as the 95th percentile.

Turn the Candidate into Something We Can Release

I fixed C’s weights, tokenizer, chat template, and serving configuration, then handed the package to the evaluation lead for the sealed acceptance test. The object being tested was this complete release. A different template or serving configuration could change what users received, so the version record had to change with it.

The sealed results were reviewed against the same requirements. After passing, the version would go to a small group of users first. A and its rollback configuration remained available. New problems, after privacy handling and review, would enter later development and regression records. Acceptance gave us grounds to open the current scope; real use would keep showing us where to improve.

Looking back at the training directory, all three files were still there, but their destinations were clear. A would provide a fallback, B would return for long-document repairs, and C would proceed through the complete checks into a trial rollout. I could finally answer the question we started with: which abilities had improved, which had been preserved, and how long users would have to wait for them.

Next, I needed to lay out the machines behind that trial window. How a question travels from the browser to a GPU, and how an answer returns piece by piece, would determine how many people the service could support.