Recursive training on synthetic data can lose distribution tails before the average collapses

Model collapse is a phenomenon where information is lost from the original distribution in a cycle where the next generation learns from the generated products of one generation, and the subsequent generation learns from those generated products. In particular, low-probability cases disappear early, and variance and quality may be impaired afterward. [1] Treating one-time data augmentation, teacher-model labeling, and mixed learning including real data all as the same "self-consumption" misidentifies the cause.

The operational question is not whether to use synthetic data, but which generator, from which real data, under what selection conditions, what part of the learning set is replaced, and over how many generations it is reused. Even with the same synthetic ratio, the recoverability of information changes depending on whether a fixed core of real data is retained or everything is replaced every generation.

This article does not address the re-identification risk when distributing synthetic data externally. It also differs in unit from the diversity of idea groups created by humans with AI assistance. The object to be protected here is the rare cases, minority classes, breadth of vocabulary and meaning in the learning distribution, and the state in which the next generation can learn them.

Shumailov et al. investigated the process of learning the next generation from the previous generation's generated data using Gaussian mixtures, variational autoencoders, and language models, both theoretically and experimentally. They reported results showing that low-probability regions of the original distribution are lost first, and errors accumulate across generations. [1] This is not research that directly observes all data on the web or current large-scale foundation models.

The language experiments are concept demonstrations centered on OPT-125m and WikiText-2, and do not reproduce the pre-training of huge commercial models. [1] Therefore, it cannot be read as "collapse becomes irreversible as soon as even one piece of synthetic data enters." What is important is the mechanism by which finite samples, model approximation, and selection during generation miss low-frequency information every generation.

This paper has an Author Correction published in 2025 to correct symbols in the theoretical section. This symbol correction changes α to β in the main text, and the publisher has corrected the HTML and PDF. [2] This is not a withdrawal of the conclusion, but one must refer to the corrected version rather than the pre-correction equations or secondary sources.

• TAIL: Check the recall of low-frequency cases and minority classes before the average loss
• GENERATION: Do not lose the generation number of synthetic items, and do not mix across generations
• REPLACEMENT: Record separately the amount of real data replaced and the fixed core retained

Kazdan et al. compared learning procedures at ICML 2025: replacing real data with synthetic data, accumulating all real and synthetic data, and using only a fixed-size subset. They reported that collapse occurs with replacement, divergence of test loss is suppressed with full accumulation, and gradual degradation remains with fixed budgets. [3]

This falsification rules out the simple conclusion that "synthetic data is dangerous and should be excluded." At the same time, it does not mean that accumulation is always safe. The experiments involved multivariate Gaussians, kernel density estimation, and fine-tuning of language models; since data duplication, rights, computational budgets, and model scales differ in actual sites, the same stability cannot be transferred. [3]

In practice, manage not only the synthetic ratio but also the absolute amount and coverage of real data. Distinguish between the result of increasing the total amount by adding synthetic data and the result of pushing out real data with the same budget. If comparison conditions are not fixed, the benefits of quantity and the loss of distributional replacement will be mixed into the same improvement rate.

Drayson et al. evaluated a method of weighting text with unknown generative origin using a detector and re-sampling human-like samples with four models, GPT-2 and SmolLM2, at EMNLP 2025. They reported conditions that prevent collapse under specific recursive learning conditions and exceed pure human data learning. [4] This is a falsification showing that synthetic samples are not always harmful and that selection can recover information.

However, detectors are not ledgers that know the origin. The research subjects were two relatively small model series and open-text generation, so misclassification will vary with unknown commercial models, translation, editing, and mixed essays. [4] Procedures that increase human-like text may also incorrectly lower atypical human expressions or minority languages.

Therefore, treat the data ID, model version, prompt, and original data reference that can be assigned at generation as the primary lineage, and place detectors as secondary estimates to supplement missing lineage. Do not delete based solely on detection probability; also record how the tails and minority groups move when thresholds are changed.

• PROVENANCE: Make the ID at generation and original data reference the primary record, and do not overwrite with detector estimates
• UNKNOWN: Place unlearned generators, translations, and edited texts in independent external tests
• GROUPS: Measure false exclusion rates by language, region, frequency, and minority class

Average loss and general benchmarks are strongly pulled toward typical examples of the majority. Even if the tails disappear, the central text may appear fluent, and overall accuracy may be maintained. To catch early signs of model collapse, tests fixed on rare words, minority classes, long-tail combinations, and creative branches are necessary. This is a measurement proposal derived from research, and no single study has established universal metrics.

We also examine other explanations. Degradation may arise not from synthetic origins but from duplication, low-quality filters, decoding temperature, teacher model bias, or proximity to the evaluator. Dichotomizing based solely on whether it is synthetic or human causes us to overlook the benefits of human data that massively replicates the same template or high-quality synthetic examples.

To separate causality, we compare conditions of original real data, added real data duplication, synthetic addition, real data replacement, and detector selection under the same computational budget. We measure not only the improvement of the first generation but also the degradation after repeating the same procedure multiple times and the recovery when returning to the original distribution.

The unit of measurement is the sample, not the file. Assign origin, generator, generation date and time, editor, original sample, selection score, and model generation used to each sample. Web-acquired data with incomplete lineage is not placed in the same column as known human or known synthetic data but is separated as unknown.

Evaluation is three-tiered. The first tier is average loss and main business quality; the second tier is minority classes, rare words, and out-of-condition cases; the third tier is duplication and semantic diversity among outputs. Furthermore, measure performance after learning by the next generation to confirm whether data that looks good in the current generation does not narrow the breadth in subsequent generations.

Do not reduce results to a single point of synthetic ratio; instead, leave them as response curves with axes of absolute amount of real data, generator, selection method, generation, and computational budget. Do not make the maximum ratio that meets the quality lower limit a permanent standard; re-estimate it if the model or domain changes.

• BASELINE: Always retain learning with the same budget using only real data
• RECURSION: Measure tails and diversity after iteration, not just in the first generation
• RECOVERY: Test the possibility of recovery by removing problematic samples and returning to the original data

Candidates for adoption are those that improve main business quality, do not break the lower limit of rare groups, and do not amplify degradation in the next generation. Even if the average improves, if one important minority group degrades, it is not offset in the overall score. We first choose reversible restrictions such as excluding only that synthetic source, lowering the ratio, or increasing the fixed core of real data.

Data volume savings, rights processing, privacy, and quality are separate gates. Suppressing model collapse does not resolve issues of personal information or copyright. Conversely, even synthetic data with privacy guarantees does not necessarily preserve the tails of the learning distribution.

Stop conditions include an increase in the rate of unknown lineage, generation-specific degradation, decreased recall of important groups, increased output duplication, and failure of detectors in external tests. To resume, one must identify the data path causing the issue, recover the fixed core, and pass re-tests across multiple generations. Synthetic data is managed not as a cost-saving measure but as an experimental asset with generations.

The versioned acceptance record separates the data generation and production path from acceptance criteria: minima for provenance completeness and recall of important groups, and maxima for output duplication and generation-by-generation degradation. A generation that breaches a criterion is quarantined, with operations returning to the last accepted version and fixed real-data core. The data owner may restart only after an independent evaluator approves multi-generation retesting against that core with the causal path removed.

• Adoption: Simultaneously meet the lower limits of average quality and tails/minority groups
• Restricted use: Exclude only the deteriorating synthetic source, generation and domain, and return to the fixed real-data core.
• Withdrawal: Stop paths where lineage cannot be traced and degradation of important capabilities after iteration cannot be localized
• Resumption: Re-learn multiple generations with corrected data and confirm recovery and recurrence prevention