[Training a 200B LLM from Scratch] 8/25: How Do We Clean the Training Data?

The previous article selected our data sources. FineWeb, FineWeb2, and the code and mathematics datasets have already done much of the preparation for us. As we combine them into our own training set, the team still has several decisions to make: which content to keep, which copy of repeated material to count, and whether damaged content can be repaired.

An already trained model can help. Letting one model read the material and judge its purpose and quality before passing suitable content to the model we are training can save considerable document-by-document review. That is how FineWeb-Edu was built. This article follows the whole cleaning process: receive existing data, use rules and models to address problems, organize the internal team and outside services, and deliver material ready for training.

Start with What Is Already in the Warehouse

Take FineWeb. Its publisher has already extracted the main text, filtered language and quality, and removed duplicates within crawl batches. We first fix the source version, inspect a small batch for text, URLs, dates, and existing scores, and connect those fields to our project.

We also retain existing quality labels. Educational-value scores and mathematics-quality scores each keep their names and scoring rules. Once sample checks confirm that they suit our purpose, they can help select material directly. The publisher's work continues to be useful.

Our running example is a constructed sorting tutorial. It asks us to keep duplicate elements. Given [3, 1, 3], sorted(xs) returns [1, 3, 3]. We will follow this tutorial into the training set.

The tutorial might appear in both a general collection and a selected collection. FineWeb-Edu is a subset of FineWeb, much like a library's complete catalog and its recommended reading list. For the same text, we retain both sources and the selection label, while counting one document in the unique inventory.

Metadata: Information about the text, such as its source, version, license record, and quality labels. It can be stored separately from the text sent to the model.

Subset: A selection from a larger collection. Its material is already included in the larger collection.

Run the Inexpensive Checks First

Programs can first check for empty records, unreadable text, and visibly broken formatting. We also handle known subset relationships between sources here. FineMath-4+ is contained in FineMath-3+, so we record that relationship when selecting material.

When different sources copy the same content, text fingerprints can identify it. Shared passages help find near-duplicate versions whose titles differ but whose main text is the same. DataTrove provides these deduplication steps. After checking, we keep a complete representative version and record the other sources.

In our sorting tutorial, “keep duplicates” and “remove duplicates” produce different results. The latter uses sorted(set(xs)) and returns [1, 3]. The wording is similar, but the teaching tasks differ, so both should remain.

Models can also help with more extensive rewrites. First, convert text into vectors that represent meaning and find candidates with similar meanings. Then compare their tasks, conditions, and answers. Similarity narrows the search; the retention decision depends on whether the content adds learning value. One symbol in code or one condition in a problem can change the conclusion.

Filtering rules also depend on the type of material. Code needs punctuation and indentation, mathematics needs formulas, and a short, complete proof can be worth keeping. When sample checks find damage to a category, we adjust its rules. An August 2026 deduplication study from the Hunyuan team likewise separates duplicate detection from retention policy and protects code structure.

Content fingerprint: A short identifier calculated from content to help find identical material quickly.

Near duplicate: Material with largely the same main content but a different title or small changes in wording.

Vector: A sequence of numbers representing features of content. Here, we use it to find text with similar meanings.

What Can Already Trained Models Take Over?

After rules handle obvious problems, some judgments remain: does this passage explain something clearly? Is the tutorial complete? Is it suitable for general reading or for teaching code? Models can help with these questions.

Start with a simple score. The FineWeb-Edu classifier is a public model with about 0.1B parameters that scores the educational value of English material. Its developers first used Llama3-70B-Instruct to score 450,000 samples, trained the small model on those examples, and then used the small model to process large volumes of text. The large model helped establish the judgments; the small model handled the repetitive work.

We can use an already trained classifier directly and first check its performance on our samples. We schedule bulk processing where the existing material lacks suitable scores. The following models have public weights as of October 2026:

The official pages for all four sets of weights list the Apache 2.0 license. The first three are specialized classification tools; Qwen is a general model that can follow instructions. NVIDIA's quality classifier also integrates directly with the September 2026 release of NeMo Curator, so existing tools can form a batch-processing pipeline.

We need to distinguish what each score means. FineWeb-Edu particularly favors basic and primary-to-secondary educational material. Research papers, advanced mathematics, and code need separate calibration. An educational score measures whether material resembles useful teaching content; a topic label identifies what it discusses. To check whether code is correct, we run it. Chinese, code, and mathematics samples also need separate calibration, rather than having all retention decisions depend on an English educational score.

For long documents, a scoring model needs to read sections and combine the results. NVIDIA's quality classifier has a context length of 1,024 tokens. Give it only the beginning of an article, and its judgment covers that beginning. We record which passages were read, sample-check whether key conditions and conclusions were covered, and choose a suitable way to combine the scores.

Classifier: A model that reads content and returns a category or score, such as “computing topic” or “high educational value.”

Calibration: Using human-checked samples to see how model scores correspond to the material we want to keep, then setting selection criteria.

Let a Large Model Read the Sorting Tutorial

For difficult material, I would give the model a specific task: “Find the tutorial's requirements, code, and expected output. Determine whether they agree. Identify the location of any problem in the original text, and recommend keeping, holding, or repairing the material.”

The complete version asks us to keep duplicate elements, uses sorted(xs), and gives [1, 3, 3] as the output. All three agree, so it can proceed to acceptance checks. If another record has the same requirement but uses sorted(set(xs)), the model should point out that this code removes duplicates.

We then run a test. Passing [3, 1, 3] to the code actually returns [1, 3], making the problem clear. We also check empty lists, negative numbers, and other cases relevant to the task. The model helps locate the error; independent checks determine whether the material can enter training.

There are three possible outcomes. Keep complete material that passes the checks. Hold material whose key conditions are missing or cannot be verified. Where there is a clear basis for repair, generate a new version and check it again.

If the original tutorial explicitly requires keeping duplicates, we can ask the model to suggest changing sorted(set(xs)) to sorted(xs), then inspect the differences and test results. If the problem statement itself is missing, a guess that “the author probably wanted to keep duplicates” provides too little evidence. That record stays on hold.

The same approach works for formatting: restore paragraph boundaries supported by the source, repair code indentation that can be checked against the original file, and organize formulas that can be compared with their source. Factual repairs require a trustworthy reference; mathematical derivations require checks of their steps and conditions. We first select material that is easy to verify and clearly valuable after repair, so the repair budget goes toward content worth keeping.

Hold: Temporarily keep questionable material out of the training manifest while preserving the original text and issue record for later review.

Save Rewrites as Separate Versions

A model can also turn fragmented material into a clear explanation or generate questions and answers from existing content. NeMo Curator's Nemotron-CC workflow includes rewriting, knowledge extraction, and question-answer generation. It provides an existing implementation of model-assisted processing to study.

We save the resulting text as a synthetic version, together with its original source, the model used, the processing instructions, and the verification results. Keeping the original lets the team see exactly what the model changed. After the new version passes content checks, we check its duplicate relationships and evaluation separation again.

For example, a model can turn the sorting tutorial into a set of questions and answers, but they still come from the same knowledge material. The original and its rewrite belong to one content family and are handled together when splitting training and test data. The new wording can be used in training experiments while its source relationship remains clear.

Articles that are already complete when received can simply be retained. Model processing takes priority where scores are missing, damage is clear, or a new form of expression is useful.

Synthetic data: Material generated or rewritten by a model. The project separately records its source, generation process, and verification results.

What Does This Actually Save?

The most direct savings are the time spent reading every document and writing judgment rules by hand. Small classifiers produce scores, while large models can understand complex requirements, explain problems, and help create the project's own labeled examples. As those examples accumulate, we can train a small classifier better suited to our material and use it for bulk scoring.

Compute costs belong in the same calculation. A large model reads the text and may generate reasons and a revised document. With API billing, count all inputs, outputs, and retries. With self-hosting, record GPU time and the amount of material completed per hour. Testing a representative batch lets us estimate the processing cost per million input tokens and how much of that material is worth keeping or repairing.

We therefore use stages: reuse suitable existing scores, handle obvious problems with inexpensive rules, apply small classifiers to appropriate bulk material, and send difficult cases to a large model. Sample the keep, hold, and repair paths separately to check both errors that survived and good material that was excluded.

Finally, run a comparison under a fixed training budget. One group uses the existing data selection; the other includes the new cleaning results. Check whether the target capabilities improve and whether capabilities we want to preserve, such as Chinese, mathematics, and code, regress. Then decide whether to expand processing. The value of cleaning shows up in what the model learns.

Threshold: The cutoff used for selection. Changing it affects both the amount retained and the pattern of mistakes.

Who Does the Work, and How Many People Do We Need?

This pipeline needs engineers and people who understand the material. Following the team plan in Part Three, we assign four internal data roles, joining as the project progresses:

These are the responsibilities of the four internal staff members. Before month 7, the first two arrivals maintain versions and counts, with the evaluation team helping validate quality. Month 2 establishes samples, standards, and trials. Batch processing begins in month 3, model-assisted processing expands in month 5, and full-batch acceptance takes place in month 7. Each expansion depends on whether throughput, retention rate, and the human-review queue can keep up with the training schedule.

The four people manage automation and delivery. Expert review and bulk annotation can be purchased by project. Vendor staffing comes from measured pilot workload: time how long each category takes to check, how much needs a second reviewer, and how much needs rework. Multiply the number of items by average work time, add review and rework time, and divide by the hours each person can devote to those tasks within the delivery period. Calculate Chinese, code, and mathematics separately; a complex proof can take much longer than ordinary text.

If machine processing is slow, improve batching or add compute. If the expert queue cannot finish on time, expand external review. If quality rules and pipeline maintenance consistently exceed internal capacity, adjust internal staffing. The bottleneck determines the headcount, and outsourced work has its own quantity and cost alongside the four internal roles.

What Can We Buy Directly?

We can buy help, and the money should address gaps in the existing material. First distinguish the purchase: a licensed collection of articles, a batch of expert demonstrations, a review service, and software that runs cleaning jobs solve four different problems. Based on official product information as of October 19, 2026, these are useful starting points:

Scale's generative AI services are particularly suited to having programmers, language specialists, and domain experts create or review demonstrations. Its Data Engine page explicitly lists question-answer generation, preference feedback, and evaluation after initial pretraining. We therefore include it in procurement for high-value expert data, while bulk pretraining text continues to come from existing corpora and licensed material.

Defined.ai is closer to selecting an existing collection: browse the catalog, obtain samples, then confirm usage rights and pricing for the selected material. Purchased articles still need cross-source deduplication against our inventory. Labelbox's role here is expert review: we supply the material and standards and receive traceable review results.

These custom services require project quotes. Scale's quote entry point leads to sales, Defined.ai offers samples and quote requests, and Labelbox expert projects are also scoped to requirements. NeMo Curator is Apache 2.0 open-source software run by our own team. Separating software, model compute, and expert fees makes total costs comparable.

We procure in five steps:

1. Describe the missing material. Include examples to keep, exclude, and debate, along with the delivery format.
2. Ask the provider for a small pilot batch, including sources, collection or generation methods, usage rights, and processing records. Specify permitted model-training and commercial uses in the contract, including the permitted uses of delivered data and trained weights.
3. Have the internal team accept or reject the batch using check samples whose answers the supplier cannot see. Check content, duplication, evaluation leakage, and retention rate. Run code tests and arrange independent review for specialist judgments.
4. Price by accepted, unique material. For text, use cost per million accepted student-model tokens; for expert work, use cost per accepted demonstration or review. Pilot costs, rejected material, compute, rework, and transfer all belong in the numerator.
5. Put acceptance standards, batch sizes, delivery dates, review methods, and replacement responsibilities into the contract, then expand batch by batch. Keep traceable records for every batch, with final acceptance owned internally.

Part Three reserved $1 million to $3 million for annual data licensing and outside processing. Pilot results will allocate that budget to specific contracts: which gaps to fill through purchases and which to address with existing data and automation. A source with low retention, heavy duplication, or frequent expert corrections can cost more per usable item even when its initial quote looks inexpensive.

Preference labeling: Comparing two responses and recording which better meets the requirements and why, for training or evaluating response quality.

Cost per accepted unit: Total cost divided by the amount of unique material that passes acceptance. Units can be student-model tokens or complete tasks; comparisons use the same standard.

Preserve the Exam and Deliver an Auditable Training Set

Before material enters the inventory, group duplicates and rewrites into families, then divide them among training, validation, and testing. Use validation data to select the cleaning approach; save the final test for after that choice. Maintain a separate exclusion list of public benchmark questions, answers, and solutions, checking both direct matches and near-duplicate leakage.

Once the selection is fixed, count it with a fixed version of the student model's tokenizer. A Chinese mathematics handout can have both “Chinese” and “mathematics” labels, but it counts only once in the total inventory. Record unique inventory separately from cumulative training reads, giving Part Seven's repetition counts a reliable denominator.

The training team receives four things:

Returning to the opening question: already trained models can certainly help clean our data. We use them to score, review, and repair, then accept the results through source checks, program tests, human sampling, and controlled training comparisons. In practice, we retain the four-person internal data team, process bulk material with existing software and models, and purchase licensed data and expert work for specific gaps. Start with samples, calculate the cost of accepted material, and expand in batches. The team keeps responsibility for source selection, quality standards, evaluation separation, and final acceptance. What reaches the training team is a dataset whose sources, quantity, and quality can all be checked.

Duplicate family: A group of copies, republications, and related rewrites of the same content.

Evaluation contamination: Test content entering training material, causing scores to overstate the model's ability to handle new problems.

Tokenizer: A tool that converts text into the token sequence the model reads. Counts use a fixed version and configuration.