A model generates examples, a filter keeps some of them and another training run begins. At each turn, a choice is made about which parts of the world the next model will see. Rare cases can disappear; useful new exercises can also be created. The argument about synthetic data becomes more informative when it follows those choices. The relevant question is what information is preserved or added, and what independent material can reveal a mistake.
Now draw a second line into the circle. It might carry original observations, retained reference data, a reliable answer checker or an evaluation designed by someone outside the generation process. That line changes the question. Instead of asking whether synthetic data is good or bad in the abstract, we can ask what evidence enters, what is retained and which mistakes the process can detect.
This distinction matters because generated data describes many different practices. Replacing an entire training collection with a model’s output is not the same intervention as adding checked examples for a narrow task. Producing alternative phrasings is not the same as inventing facts. A useful account names the workflow before drawing a conclusion about its likely result.
The part of the distribution that disappears first
The 2024 Nature paper AI models collapse when trained on recursively generated data examined degradation under recursive use of generated material and described the loss of tails of the original distribution. Its results are a serious warning about the conditions studied. The paper has a March 2025 correction to a parameter label in its theoretical-intuition section; that is distinct from a retraction of its reported result. The warning should not be translated into a claim that any use of any generated example immediately ruins a model.
The tail is worth explaining without anthropomorphism. A distribution can contain common cases and rare cases. A generator sampling from its current approximation may supply many examples of the common cases while missing some rare ones. If the next training stage learns from that reduced view without another source of information, the omitted cases can become harder to recover.
Consider a deliberately simple mathematical example. An independent draw has a one-percent chance of producing a rare category. In one hundred draws, the probability of seeing none of that category is 0.99 to the hundredth power, about 36.6%. These are assumed probabilities in a sampling model, not measured frequencies from a language model or a particular training corpus.
If a toy learner merely copies empirical frequencies and assigns zero probability to categories absent from those samples, it will lose the rare category on those runs. That learner is much simpler than a modern language model. The example illustrates how a missing tail can arise through finite sampling; it does not establish the speed or inevitability of collapse in a real training pipeline.
A rare category is not necessarily unimportant. It might represent an unusual expression, a less common task or an exception that matters disproportionately to a user. Optimizing an average score dominated by frequent cases can conceal that loss. A team needs to know which parts of the intended task distribution it must preserve, rather than assume frequency and value are interchangeable.
Replace the archive, or add to it?
The paper Is Model Collapse Inevitable?, introduced in 2024, examined a different data workflow: accumulating generated material alongside retained real data. Its experiments found that retaining and accumulating data could avoid the collapse observed under replacement in the settings studied. That result identifies an important design distinction, not an unlimited guarantee that an ever-growing archive solves every data problem.
The distinction is concrete. Replacement asks a later generation to learn from what the previous model reproduces. Accumulation can keep earlier evidence available even when a later sample fails to reproduce it. The retained evidence may help preserve information that would otherwise be absent from the next training collection. Whether the training actually uses it effectively is another question.
Storage and training budgets complicate the picture. A team might retain an archive but sample only a fixed amount from it. The retained data then exists without necessarily being seen often enough to have the intended effect. The relevant record should describe sampling and weighting, not simply announce that original data has been kept somewhere.
Collapse or Thrive?, first submitted in 2024 and revised in 2025, makes those distinctions testable. It examined Gaussian estimation, kernel density estimation and language-model fine-tuning, with three different data workflows:
| Workflow studied | What the authors reported in those settings |
|---|---|
| Replace real data with successive synthetic generations | Collapse across the tested task settings |
| Accumulate and train on all combined real and synthetic data | Stable models without diverging test loss |
| Accumulate the archive but train on fixed-size subsets | Slow, gradual degradation across generations |
The final row matters to a team with a finite compute budget. Keeping an original archive is not the same intervention as training on the entire accumulated archive at every generation. The reported outcomes belong to these settings; they do not supply a universal threshold for a modern production model. They do explain why two papers about synthetic data can reach different conclusions without describing the same experiment.
Retained data also needs a quality account. Human-produced does not mean error-free, representative or appropriately licensed for every activity. An old archive can contain systematic gaps that recursion preserves rather than fixes. Adding external evidence is valuable when it improves the connection to the intended task, not because a human origin automatically makes every entry authoritative.
A generated example can still teach something useful
Instruction-tuning research provides a different use case from wholesale replacement of pretraining data. Self-Instruct, first submitted in 2022 and published at ACL 2023, generated instructions and examples, filtered unsuitable or overly similar material and used the result for tuning. It reported gains in the evaluations it performed. Its pipeline demonstrates why selective generated data and uncontrolled recursive replacement should not be treated as identical experiments.
The potential benefit is understandable. A model may be able to express an existing task in many useful forms, supply examples that follow a known specification or generate practice material for a capability that can be checked. The generated examples can make some aspects of training more accessible. They do not necessarily introduce new facts beyond what the generator and its inputs already contain.
Suppose a fictional training team wants a model to convert short inventory notes into a fixed set of fields. Generated notes could vary word order, omissions and harmless distractors while a controlled template supplies the expected fields. That is a bounded design. The team still needs to test notes written independently, because template-generated diversity may fail to resemble the variation found in actual use.
The checker should match the intended task. Valid JSON proves that an output follows a syntax, not that every extracted value is correct. A program passing a limited test suite proves something about those tests, not all possible inputs. Verification can be powerful while remaining partial, and training records should not promote one kind of check into a broader claim.
What gets filtered out deserves inspection
A filtering stage can improve a generated collection by removing invalid or redundant material. It can also shape the collection in ways the team did not intend. If the filter prefers fluent, familiar responses, unusual but correct examples may be rejected more often. An apparently cleaner dataset can become narrower without anyone explicitly deciding to reduce its coverage.
Inspect the rejected material as a sample, not only the accepted set. Ask which categories are excluded and whether that follows the task definition. A high acceptance rate says little about quality if the checker misses a recurring error. A low acceptance rate can be worthwhile if it selects useful cases, but generating large volumes does not make the selection criterion correct.
The generator and judge may share weaknesses. Two models agreeing is evidence of agreement, not automatically independent confirmation. If both have encountered similar material or learned similar shortcuts, they can endorse the same error. A genuinely useful check may need an external record, a task-specific procedure or human expertise appropriate to the claim being evaluated.
Human review has limits too. Reviewers can be inconsistent, miss subtle errors or adopt the generator’s framing. The response should be a defined review process with examples, disagreement handling and appropriate scope, rather than an unexamined assumption that adding a person makes every label reliable. The objective is an evidence chain the team can inspect.
Keep the examiner outside the lesson plan
Evaluation data should test the intended capability without merely repeating the generator’s preferred wording. If the same templates create both training and test examples, a system may learn the templates while missing the broader task. Holding out some generated rows is not necessarily enough when those rows share the same underlying construction.
For the fictional inventory task, an independent test could include real authorized examples or separately authored notes reflecting the intended use. It should contain relevant exceptions, missing information and cases the system ought to route for clarification. The test needs an account of its own coverage and privacy conditions. Independence is a design property to establish, not a synonym for a different filename.
Examine categories as well as averages. A model can improve common cases while losing rare ones, or become more confident on inputs where the correct behavior is to decline. Which trade-offs are acceptable depends on the application. A single aggregate gain should not silently authorize a regression in a category the service has promised to handle.
Keep the evaluation stable enough to compare generations, while adding a separate route for new evidence. Repeatedly rewriting the test to resemble the latest model’s strengths would remove the outside check the process needs. Conversely, freezing an obsolete task definition forever can make the score irrelevant. Documented revisions preserve the distinction between maintaining a test and selecting a favorable result.
The record that should accompany the next model
An informative training record identifies the source materials, generator, selection rules, retained reference data and relevant sampling choices. It should distinguish observed labels from generated ones where that matters and connect the accepted model to the evaluation performed. This record makes it possible to investigate why a later generation changed, rather than infer the cause from a broad label such as synthetic-heavy.
Cost should include checking and failed generation, not only the price of accepted tokens. A cheap generator can produce expensive review work if errors are subtle. A more structured process may generate fewer examples that teach the intended behavior more reliably. Volume is an input measure; the resulting capability still needs to be demonstrated.
Count distinct information as carefully as rows. A hundred variations of one seed example can be valuable practice without supplying a hundred independent observations about the world. If those variations retain the same incorrect assumption, the apparent size of the collection can make that assumption look unusually well supported. Deduplication and diversity checks therefore need to consider meaning and construction, not just identical text. The record should allow a reviewer to identify families of examples and inspect their common origin. That makes it easier to correct a flawed seed and understand how far its influence spread through the accepted material.
The circle on the page can now be redrawn as a workflow with explicit decisions. What is generated? What survives? What external evidence remains available? Who or what checks the result, and what cases stay outside the teaching loop? Those questions are less dramatic than predicting inevitable collapse or effortless self-improvement. They are also the questions a team can answer before asking a model to teach its replacement.
Sources & context
This article draws on the sources linked in the text. Analysis and illustrative scenarios are identified separately from reported findings.
- AI models collapse when trained on recursively generated data
- March 2025 correction
- Is Model Collapse Inevitable?
- Collapse or Thrive?
- Self-Instruct
